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I. 


INTRODUCTION 


An  automated  Air  Combat  Maneuvering  Performance  Measurement  (ACMPM) 
system  should  be  invaluable  for  the  assessment  or  improvement  of  air¬ 
crew  combat  readiness.  Because  of  the  complex,  dynamic  and  fast 
moving  nature  of  the  air  combat  task,  present  assessment  techniques 
based  on  engagement  outcome  and  instructor  recall  of  a  few  of  the 
characteristics  of  the  engagement  are  inadequate.  A  good  ACMPM  system 
would  provide  a  detailed,  standardized  appraisal  of  pilot  performance 
that  is  not  currently  available. 

Purposes  of  ACMPM  Measurement 

Information  derived  from  such  an  ACMPM  system  would  be  used  for 
several  major  purposes.  First,  it  would  provide  improved  feedback  to 
Air  Combat  Maneuvering  (ACM)  students  concerning  their  progress  through 
the  flight  training  curriculum.  It  also  would  provide  diagnostic  in¬ 
formation  to  each  pilot,  indicating  his  areas  of  individual  strength  and 
weakness  in  the  performance  of  ACM  tasks.  Second,  it  would  provide  ACM 
instructors  with  better  information  about  their  students'  progress, 
allowing  the  instructors  to  adapt  the  training  program,  within  given 
constraints,  to  the  needs  of  the  individual  student.  It  also  might 
provide  instructors  with  diagnostic  information  about  their  own  perfor¬ 
mance  in  teaching  ACM  if  they  can  see  a  pattern  of  strengths  and 
weaknesses  in  their  students. 

A  third  function  of  an  automated  ACMPM  system  would  be  to  provide 
feedback  to  instructional  system  development  (ISD)  personnel  concerning  the 
efficacy  of  current  training  materials  and  syllabi.  Consistent  patterns 
of  weakness  in  the  students  would  serve  as  an  indicator  of  a  need  for 
adjustment  and  improvement  in  the  program.  In  areas  where  students 
consistently  demonstrated  much  higher  than  expected  performance,  train¬ 
ing  emphasis  could  be  decreased  profitably. 

A  fourth  use  of  an  automated  ACMPM  system  would  be  to  provide  better 
tactics  assessment.  When  new  tactics  are  devised  and  tested,  a  good 
ACMPM  system  would  improve  the  speed  and  accuracy  with  which  the  results 
could  be  evaluated.  Finally,  an  ACMPM  system  could  provide  accurate, 
standardized  measures  of  the  operational  readiness  of  pilots  and  of 
units. 


The  most  crucial  aspect  in  the  development  of  an  automated  ACMPM 
system  is  in  the  choice  of  measures  to  be  taken.  The  measures  must  pro¬ 
vide  a  valid  and  reliable  assessment  of  pilot  performance.  This  means 
that  a  highly  skilled  pilot  consistently  must  achieve  a  higher  score 
than  a  less  skilled  pilot.  The  measurement  system  must  be  usable  by  IPs 
and  the  results  must  be  consistent  with  their  expert  judgment.  Also,  in 
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order  to  provide  maximum  utility,  the  measurement  system  should  be 
diagnostic  of  performance  '^oficiencies  and  deficiencies.  It  must  pro¬ 
vide  an  indication  of  why  tne  pilot  achieved  a  given  score  rather  than 
just  providing  a  single  score  related  to  the  pilot's  overall  performance. 

Study  Goals 

The  goal  of  this  study  was  to  develop  a  preliminary  measurement 
structure  and  measure  set  fcr  an  automated  ACMPM  system  which  cculd  be 
implemented  o"  the  Simulator  for  Air-to-Air  Combat  (SAAC)  at  Luke  AFB, 
Arizona.  A  fully  developed  aC .y,V  system  should  be  expected  to  provide 
valid,  diagnostic  measurement  1 nfo^na^’on  in  near  real  time  for  the  use 
of  the  ACM  instructor.  The  system  ii  ••u tended  to  be  used  to  assess 
pilot  performance  during  one-vt*sun-cne  OvI)  free  engagements. 

Although  the  measurement  structure  s-v!  measure  set  were  to  be  specifi¬ 
cally  designed  for  the  SAAC:  it  was.  trie  goal  of  the  investigators  to 
provide  the  basis  for  a  system  that  could  be  implemented  in  a  variety  of 
ACM  training  settings  such  the  Air  Combat  Maneuvering  Range  (ACMR) 
and  Air  Corncac  Mc-neuverinn  Tn<?trumentation  'ACM;). 

[A  jerr.auivg  Approaches  to  the  Problem 

three  aUernative  appr. aches  to  tn-i  development  of  such  an  ACMPM 
system  are  pcsv'tie,  !a)  jure'y  .lytical,  (b)  purely  empirical,  and 
(c)  combi raa  analytica  l  ar  c.’  enr;> ; ,  a  cal . 

Purely  Analytical  Approach.  Curing  the  development  of  new  training 
systems,  the  familiar  I5D  process  has  been  successful.  The  process 
builds  hierarchies  of  intermediate  training  and  performance  objectives 
(and  criteria)  through  an  analysis  of  the  task  and  training  requirements. 
Usually,  expert  judgment  determines  what  the  intermediate  performance 
objectives  and  criteria  should  be.  Although  this  works  well  for  conven¬ 
tional  task  training,  it  might  be  a  mistake  to  rely  completely  on  an 
ISD  analysis  for  ACM  free  engagements. 

ACM  may  be  viewed  as  a  complex,  but  unitary,  task  in  itself.,  with 
variah>  a;tion  requirements.  The  more  complex  and  variable  the  task, 
the  greater  will  be  the  peril  that  the  long  chain  of  assumptions  and 
inferev  ?s  (inherent  in  the  analysis  process)  will  lead  to  incorrect 
conclusions  about  performance  criteria.  Therefore,  if  the  ACMPM  system 
were  developed  by  a  purely  analytic  process,  the  chances  of  producing 
a  successful  measurement  system  would  be  minimal. 

Purely  Empirical  Approach.  The  second  alternative  is  to  record  a 
'large  amount  of  comprehensive  data  on  ACM  engagements,  subdivide  the  data 
according  +'  outcome  success  and  expert  judgment  of  skill,  and  then 
develop  e  -,.:rical  relationships  between  the  recorded  performance,  out¬ 
come  and  judged  skill  level.  The  list  of  variables  would  be  large  and 
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unselected.  One  could  record  the  time  histories  of  these  variables 
and  perform  multiple  regression,  multiple  discriminant  or  factor 
analyses  to  determine  the  relationships  between  these  variables  and  out¬ 
come  success  and  judged  skill  level. 

Without  some  initial  rules  for  the  selection  of  interpretable  mea¬ 
sures,  the  quantitative  relationships  that  might  be  derived  from  this 
approach  would  be  of  questionable  diagnostic  value.  For  example,  it 
might  be  difficult  to  derive  useful  training  and  performance  diagnosis 
information  from  a  measure  such  as  average  g's  during  a  whole  engagement. 
It  is  more  useful  to  know  if  the  student  used  his  available  g's  when  he 
should  have,  and  did  not  waste  energy  pulling  g's  when  it  was  not 
necessary.  A  set  of  comprehensive  relationships  such  as  these  is 
difficult  to  define  using  purely  an  empirical  approach. 

Combined  Approach.  A  combined  analytic  and  empirical  approach  to 
the  development  of  useful  aircrew  performance  measurement  has  been 
shown  to  be  effective  by  several  researchers  (e.g.,  Vreuls  &  Wooldridge, 
1977;  Waag  &  Knoop,  1977).  In  this  approach,  an  analysis  is  performed 
to  define  candidate  tasks  and  measures  of  importance,  data  are 
collected  using  these  measures,  and  empirical  analyses  are  performed  on 
the  data  to  determine  the  relative  importance  of  the  candidate  measures. 
Performance  measures,  instructor  ratings,  and  measures  of  outcome  may 
be  included  in  the  candidate  set  so  that  the  empirical  analysis  may 
find  the  functional  relationships  between  them. 

Although  the  emphasis  of  the  approach  is  empirical,  it  does  not  mean 
that  task  and  training  analysis  is  unnecessary.  On  the  contrary,  the 
combined  approach  allows  the  performance  and  training  analysis  to  be 
conducted  with  a  wider  scope  because  there  are  few  initial  restrictions 
on  the  variety  of  variables  that  can  be  considered  as  potentially 
important  for  a  performance  measurement  system.  In  effect,  variables 
which  may  appear  to  be  of  marginal  importance  can  be  included  in  the 
list  of  candidate  measures.  There  are  restrictions,  however,  in  the 
number  of  measures  that  can  be  resolved  by  empirical  analysis  and  for 
this  reason  one  does  not  suggest  candidate  measures  indiscriminantly. 
Nevertheless,  the  task  and  training  analysis  should  be  as  thorough  as 
possible  to  ensure  that  all  important  variables  are  included  in  the  data 
collection,  even  those  which  might  be  excluded  in  a  more  conventional 
approach  to  the  development  of  performance  measurement  systems. 

The  initial  analysis  provides  understanding  and  insight  into  the  way 
ACM  performance  is  conceptualized  by  experts  intimately  involved  in  it. 
These  insights  are  of  critical  importance  for  determining  how  the  data 
collection  is  designed,  and  how  the  empirical  data  will  be  handled 
during  subsequent  analysis. 


Each  of  these  three  alternative  measurement  approaches  has  certain 
advantages  and  disadvantages.  Each  may  find  favor  in  certain  applica¬ 
tions.  If  the  task  under  scrutiny  is  relatively  simple  and  straight¬ 
forward  or  if  no  performance  data  are  available,  the  purely  analytic 
approach  may  be  the  method  of  choice.  If  large  quantities  of  data  and 
data  processing  capability  are  available  and  measurement  results  need 
not  be  interpreted  by  non-scientific  personnel,  the  empirical  approach 
may  be  used.  If  data  processing  capabilities  are  limited  and  the 
results  will  be  used  by  operational  personnel,  the  combined  method  may 
provide  the  best  results.  Thus,  the  choice  of  a  performance  measure¬ 
ment  approach  depends  largely  on  the  specific  application  for  which  it 
will  be  used. 

Technical  Approach 

A  combined  analytical  and  empirical  approach  was  used  to  identify 
those  measures  which  would  highlight  the  differences  in  performance 
between  Replacement  Training  Unit  pilots  and  experts  during  one-versus- 
one  ACM  free  engagements.  The  project  was  conducted  in  three  distinct 
but  interconnected  phases:  Phase  1,  problem  analysis;  Phase  2,  data 
collection;  and  Phase  3,  empirical  data  anlaysis.  These  phases  are 
summarized  below  to  provide  an  overview  of  the  contents  of  this  report. 

Phase  1.  Phase  1  was  an  analytical  examination  of  the  ACM  task  for 
the  purpose  of  developing  candidate  measures.  Information  was  obtained 
from  two  main  sources, (a)  ACM  training  materials  and  (b)  interviews 
with  subject  matter  experts  including  ISD  and  academic  personnel,  ACM 
instructors,  and  Fighter  Weapons  School  personnel.  From  this  information 
hypotheses  were  developed  about  the  structure  of  ACMPM  algorithms  and 
about  important  parameters  to  be  included  in  this  structure. 

Phase  2.  Phase  2  involved  collection  of  empirical  data  on  ACM  in 
the  simulator  for  Air-to-Air  Combat  (SAAC)  at  Luke  AFB.  Data  collection 
lasted  for  five  weeks  during  which  a  total  of  405  engagements  were  flown. 

Each  week  a  total  of  six  F-4  pilots  of  varying  training  and  experi¬ 
ence  levels  took  part  in  data  collection  over  a  period  of  three  days. 

The  first  day  of  data  collection  during  each  week  was  devoted  to 
determining  the  entry  skill  level  of  the  participants.  After  filling 
out  a  background  questionnaire,  the  pilots  flew  a  set  of  pretest 
exercises.  These  exercises  consisted  largely  of  the  Vought  Corporation 
Good  Stick  Index  (GSI)  exercises  (USAF,  1977).  Each  participant  flew 
five  attacks  on  a  non-reactive  target  flown  on  instruments  by  a  project 
pilot.  Three  of  these  were  head-on  attacks,  one  was  a  cine-tracking 
exercise,  and  one  required  a  missile  shot  and  a  high  deflection  gun 
pass.  These  exercises  were  chosen  because  they  represented  an  existing 
test  of  basic  ACM  skill  level  and  because  they  would  allow  comparison 
of  this  data  base  with  the  existing  Vought  Corporation  data  base. 
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The  second  and  third  days  of  data  collection  each  week  consisted  of 
a  round  robin  series  of  engagements  among  the  six  participants.  Every 
pilot  flew  against  each  of  the  other  pilots  in  the  group  three  times  in 
F-4  versus  F-4  competition.  Three  initial  setups  were  used  in  which 
neither  pilot  had  an  advantage.  On  the  third  day  of  competition  the 
two  most  experienced  ACM  pilots  flew  six  engagements  of  dissimilar  air 
combat  with  the  simulator  configured  as  F-4  versus  MIG  21. 

Data  were  recorded  on  67  different  variables  for  each  cockpit. 

These  included  measures  of  relative  aircraft  position,  energy  states, 
weapons  switchology,  and  control  position  and  inputs.  As  another 
measure  of  pilot  performance,  the  project  pilots  who  were  experienced 
ACM  instructors  observed  each  engagement  and  at  the  end  of  each  day 
rated  the  participants  on  such  characteristics  as  aggressiveness, 
situation  awareness,  application  of  basic  fighter  maneuvers  (BFM)  and 
overall  ACM  ability.  Finally,  afte.-  their  last  engagement,  each  of 
the  participants  were  asked  to  rank  order  ail  of  the  pilots  in  their 
group  in  overall  ACM  performance.  The  pro jp'*'.;  pilots  also  completed 
this  ranking. 

Phase  3.  In  Phase  3  an  analysis  of  the  Phase  2  data  was  performed. 

The  engagements  analysis  data  base  contained  well  over  19  million  data 
points  (67  parameters  for  each  of  two  cockpits,  sampled  an  average  of 
360  times  during  405  engagements),  exclusive  of  control  stick  measures 
(which  were  sampled  an  average  of  1800  times  per  engagement). 

The  data  analyses  were  performed  at  three  different  levels  of 
complexity,  (a)  definition  of  skill  level,  (b)  engagement  success  versus 
skill  level  and  (c)  tactics  versus  skill  level. 

The  first  analysis,  definition  of  skill  level,  was  oerformed  to  insure 
that  the  initial  assignment  of  pilots  into  groups  of  different  skill  levels 
was  valid,  because  all  subsequent  analysis  of  data  would  presuppose 
that  Group  1  pilots  were  in  fact  less  skilled  than  those  in  Group  2  or 
3.  Sources  of  data  included  pilot  background  questionnaires,  pretest 
performance  peer  ratings,  project  pilot  ratings  and  actual  engagement 
outcomes.  The  ejgagement  success  versus  skill  level  analysis  probed 
the  data  for  measures  which  could  be  used  for  an  entire  engagement. 

Also,  the  analysis  asked  (a)  whether  the  three  setup  conditions  were 
equivalent,  i.e.,  could  they  be  collapsed  to  increase  the  replications, 
and  (b)  whether  the  pretest  performance  was  predictive  of  free-engage- 
ment  performance,  i.e.,  could  the  canned  pretest  be  used  as  a  skill 
indicator. 


The  kind  of  measures  which  were  characteristic  of  engagement  level 
data  included  the  following:  (a)  engagement  outcome  (win,  lose  or  draw); 
(b)  the  percentage  of  time  each  pilot  was  in  an  offensive  mode  with 
advantage  (low  aspect  and  angle  off),  offensive  with  no  advantage  (head- 
on  gun  pass),  in  a  neutral  position,  or  at  a  disadvantage,  and  (c) 
global  measures  such  as  fuel  flow,  airspeed,  throttle  position,  speed 
brake  deployment  and  the  amount  of  time  the  opponent  was  in  visual 
contact. 

It  is  important  to  note  that  the  data  for  an  individual  pilot  can¬ 
not  stand  alone.  raJ.  ^-gagement  had  to  be  viewed  as  a  confrontation 
between  a  pair  O'i  pilots.  T’h'is,  the  analysis  had  to  indicate  both  the 
metrics  of  perf". manrj  and  the  relative  skill  level  of  the  opponent. 

A  final  tactics  versus  skill  level  analysis  was  an  attempt  to  explore 
diagnostic  measurement  --  measures  which  could  aid  the  instructor  in 
determining  why  a  student  won  or  lost  an  engagement.  The  approach  to 
the  tactics  analysis  sought  a  method  of  describing  what  happened  within 
an  engagement  that  reflected  pilot  tactics,  but  did  not  require  the  use 
of  classic  BFM  profile  matching.  The  approach  also  sought  a  way  to 
partition  a  whole  engagement  into  factors  within  the  engagement  that 
might  otherwise  cause  variability  at  the  engagement  level  of  analysis. 

An  analysis  of  the  BFM  and  ACM  curriculum  revealed  that  students 
were  taught  to  look  for  relative. positions  and  rates  of  change  of  the 
two  aircraft  involved  in  an  engagement.  Rules  for  each  possible  combin¬ 
ation  of  positions  and  closure  rates  are  taught.  Since  relative 
positions  and  rates  of  the  two  aircraft  were  sampled  every  half-second, 
it  was  possible  to  subdivide  the  data  according  to  the  values  of  these 
variables  over,  a  whole  engagement.  This  kind  of  structure  is  directly 
related  to  the  way  ACM  is  taught  and  should  have  meaning  to  instructors 
when  the  format  is  explained. 

One  (of  many)  oossible  subdivisions  was  a  three  dimensional  matrix 
formed  by  sight  angle,  aspect  angle  and  slant  range.  The  matrix  thus 
formed  was  called  TACSPACE  (or  Tactical  Space),  and  it  contained  125 
"boxes"  which  were  defined  by  all  possible  combinations  of  five 
different  values  of  sight  angle,  aspect  angle  and  slant  range.  The 
measures  which  had  been  defined  for  the  engagement  level  (i.e.,  fuel  flow, 
aircraft  attitudes  and  rates,  altitude,  speed,  etc.)  were  placed  in 
each  box.  In  addition,  calculations  were  made  on  the  amount  of  time 
spent  in  each  box  and  the  number  of  times  the  box  was  entered.  It  was 
assumed  from  the  onset  that  125  cells  in  the  TACSPACE  matrix  would  be 
too  many  for  a  finalized  measurement  structure.  For  initial  research, 
however,  it  was  necessary  to  form  this  many  cells  to  explore  the  perfor¬ 
mance  space  and  to  provide  data  which  would  support  simplification. 


Final  Comment  on  Technical  Approach.  Frequently,  it  is  asked: 

Why  is  it  useful  to  have  several  measures  of  ACM  performance  when  the 
ultimate  criterion  is  the  outcome  of  the  figh'?  Certainly,  the  outcome 
is  important;  but,  as  discussed  in  Section  II,  '•here  are  many  attributes 
of  ACM  performance  that  contribute  to  the  outcome.  These  intermediate 
performance  factors  provide  information  which  can  be  used  (a)  to  diag¬ 
nose  why  a  particular  outcome  occurred,  and  (b)  to  guard  against  the 
reward  of  behavior  that  might  be  simulator  specific  and/or  inappropriate 
in  the  air.  For  these  reasons,  many  measures  are  sought  to  capture 
intermediate  performance  (as  described  in  Sections  III  and  IV),  and  the 
whole  approach  is  guided  by  the  assumption  that  the  results  of  this 
study  must  be  validated  in  the  air. 


II.  PHASE  1:  PROBLEM  ANALYSIS 


During  Phase  1  an  analysis  was  performed  to  develop  information 
which  would  lead  to  the  specification  of  a  candidate  set  of  performance 
measures  and  an  algorithmic  structure  to  use  those  measures.  The 
analysis  was  guided  by  our  past  experience  in  performance  measurement 
development  and  provided  a  framework  for  the  empirical  research  during 
Phases  2  and  3.  The  results  of  this  analysis  are  summarized  below  in 
terms  of  (a)  current  training  and  evaluation,  (b)  possible  future 
training  and  evaluation,  (c)  elements  of  pilot  ACM  performance, 

(d)  existing  ACM  models, and  (e)  the  technical  challenge. 

Current  Training  and  Evaluation 

Students  in  F-4  Replacement  Training  Unit  (RTU)  training  at  Luke  AFB 
begin  their  curriculum  with  basic  flight  and  instrument  training.  They 
then  begin  training  in  Basic  Fighter  Maneuvers  (BFM)  where  they  are 
taught  maneuvers  to  be  used  in  attacking  a  target  flying  a  known, 
predictable  profile.  BFM  training  involves  a  single  attacking  aircraft 
and  a  single  target  aircraft  and  is  used  as  an  introduction  to  ACM. 

Training  in  ACM  involves  multiple  aircraft  attacks  on  a  counter- 
offensively  maneuvering  target.  This  prepares  the  student  for  the  kind 
of  multiple  aircraft  engagements  envisioned  in  current  tactical  doctrine. 

Much  of  the  academic  and  practical  portion  of  BFM  and  ACM  training 
involves  learning  the  rules  about  proper  responses  to  various  adversary 
maneuvers.  BFM  students  learn  to  perceive  the  aspect  angle,  angle-off, 
and  closure  rate  of  the  opposing  aircraft.  They  learn  the  proper 
maneuver  for  each  possible  combination  of  these  three  variables  in 
order  to  reduce  aspect  and  angle-off  and  to  optimize  closure  rate  or  to 
maintain  a  desirable  offensive  position. 

ACM  training  involves  the  same  kinds  of  rules.  One  major  difference, 
however,  is  that  the  opponent  is  maneuvering  unpredictably.  For  this 
reason,  the  appropriate  action  usually  changes  before  a  maneuver  is 
completed  and  the  ACM  pilot  must  change  to  a  different  maneuver.  A 
successful  ACM  attack  usually  involves  initiating  portions  of  several  of 
the  BFM  maneuvers  rather  than  one  single  completed  maneuver. 

Evaluation  of  BFM/ ACM  performance  during  training  is  almost  totally 
subjective.  At  the  completion  of  a  maneuver,  the  instructor  provides  a 
score  of  0,  1,  2,  3,  or  4  based  on  performance  of  the  maneuver  as  a 
whole.  The  standards  for  assessment  of  the  score  provided  by  the 
instructor  are: 
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0  -  Indicates  a  lack  of  ability  or  knowledge. 

1  -  Performance  is  safe  but  indicates  limited  proficiency. 

Makes  errors  of  commission  or  omission. 

2  -  Performance  is  essentially  correct.  Recognizes  and 

corrects  errors. 

3  -  Performance  is  correct,  efficient,  skillful  and 

without  hesitation. 

4  -  Performance  reflects  an  unusually  high  degree  of 

ability. 

Many  instructors  indicated  dissatisfaction  with  this  method  of 
assessing  pilot  performance,  stating  that  not  enough  anchoring  was  pro¬ 
vided  to  give  them  a  good  idea  what  each  score  meant.  They  felt,  for 
example,  that  they  could  tell  when  a  student  was  flying  a  maneuver 
deserving  a  score  of  2,  but  could  not  readily  verbalize  why  a  score  of 
2  was  given.  They  indicated  a  need  for  a  more  useful  and  diagnostic 
grading  system. 

Possible  Future  Training  and  Evaluation 

In  an  attempt  to  eliminate  many  of  the  problems  in  current  training 
and  evaluation  methods,  the  Fighter  Weapons  School  (FWS)  at  Nellis  AFB 
is  developing  an  experimental  program  to  train  ACM  pilots.  Although 
students  come  into  this  program  with  varying  amounts  of  ACM  training 
and  experience,  all  students  are  started  at  the  beginning  of  the  program 
and  work  through  the  syllabus  at  their  own  speed.  This  is  in  marked 
contrast  to  th*.  traditional  curriculum  which  prescribes  a  given  number 
of  hours  and  sorties  to  train  each  phase  with  little  flexibility  in  the 
training  times. 

In  contrast  to  the  usual  0  to  4  scoring,  all  evaluation  is  on  a 
criterion  referenced  pass-fail  basis.  Student  advancement  is  determined 
\j  performance  while  flying  specified  maneuvers  against  a  predictable 
target.  If  he  is  successful,  the  student  advances  to  the  next  step  in 
the  syllabus;  if  not,  he  remains  in  the  phase  until  he  can  pass  the  test. 
This  appears  to  provide  a  promising  advance  in  training  and  evaluation 
methods  and  may  provide  a  basis  for  the  syllabus  of  future  ACM  training 
at  the  RTU  level. 

It  is  important,  to  note  that  FWS  personnel  have  recognized  their 
measurement  problem.  They  have  taken  the  approach  of  carefully  con¬ 
trolling  many  setups  and  required  maneuvers,  then  measuring  the  outcome. 
The  setups  are  organized  so  that  the  trainee  must  demonstrate  proper 
skill  in  (a)  aircraft  handling,  (b)  situation  awareness  and  (c) 
assessment  of  turning  room  to  be  able  to  arrive  in  a  firing  position  at 
the  end  of  the  maneuver.  Given  a  carefully  conceived  gradation  of  setup 
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exercises,  the  failure  of  a  student  to  arrive  in  a  firing  position  pro¬ 
vides  an  avenue  for  performance  diagnosis.  This  is  one  approach  to 
improving  performance  information.  Another  approach  is  to  try  to 
empirically  measure  what  the  two  aircraft  are  doing.  FWS  personnel 
recognize  the  potential  power  of  the  ACMI  for  training  performance 
measurement.  However,  FWS  has  yet  to  develop  ACMI  for  routine  training, 
probably  because  the  job  is  not  a  trivial  one. 


Elements  of  Fighter  Pilot  ACM  Performance 

In  order  to  obtain  a  list  of  possible  measures  of  ACM  performance 
which  were  sensitive  to  skill  level,  a  number  (7  in-depth  and  15-20 
informally)  of  Air  Force  RTU  Instructor  Pilots,  Fighter  Weapons  School 
Instructor  Pilots,  and  ISD  personnel  were  interviewed.  These  experts 
were  questioned  about  current  methods  of  assessing  performance  on  BFM 
and  ACM  in  their  units.  They  were  asked  what  characteristics  distinguish 
the  high  skill  level  ACM  pilot  from  the  low  skill  level  ACM  pilot  and 
how  these  characteristics  can  be  observed  and,  hopefully,  measured 
during  an  ACM  engagement.  From  these  interviews,  a  number  of  potential 
measures  which  may  differentiate  between  fighter  pilot  skill  levels 
were  isolated. 

Wins  Engagements.  The  most  intuitively  obvious  measure  of  fighter 
pilot  skill  level  is  the  number  of  kills  achieved  by  the  pilot  in  real 
or  simulated  ACM  engagements.  It  is  natural  to  assume  that  the  better 
pilot  will  achieve  more  kills  and  win  more  engagements  than  will  the 
less  apt  pilot.  While  this  measure  is  probably  the  most  common  cri¬ 
terion  in  use  today,  several  problems  are  inherent  in  its  employment. 

In  the  modern  air  combat  arena,  sophisticated  electronic  equipment 
has  eliminated  much  of  the  need  for  the  kind  of  maneuvering  associated 
with  the  classic  ACM  engagement.  During  the  recent  conflict  in  South¬ 
east  Asia,  a  high  percentage  of  the  kills  recorded  by  both  sides 
involved  relatively  long  range  missile  shots  with  little  or  no  high  per¬ 
formance  maneuvering  taking  place.  The  level  of  ACM  skill  of  the  pilots 
involved  was  only  a  minor  factor  in  the  outcome  of  these  engagements. 

Now,  with  the  advent  of  the  all-aspect  missile, pilot  skill  in 
high  performance  ACM  may  become  even  less  a  factor  in  outcomes  of 
actual  aerial  engagements. 

During  actual  combat,  kills  may  be  a  relatively  low-frequency 
occurrence.  When  dealing  with  such  low-frequency  events,  chance  plays 
an  inordinately  large  role.  Given  a  group  of  pilots  with  identical 
skill  levels  and  similar  opportunity,  there  will  be  large  differences 
in  the  number  of  kills  obtained,  strictly  as  a  result  of  chance. 


During  simulated  ACM,  the  number  of  kills  may  also  provide  an 
unrealistic  assessment  of  the  pilots'  potential  for  achieving  kills  in 
the  combat  environment.  Such  measures  taken  during  simulated  combat 
are  contaminated  by  the  absence  of  any  real  hazard.  Simulated  ACM, 
therefore,  does  not  account  for  differences  in  the  potentially  important 
factor  of  risk-taking  behavior. 

This  discussion  does  not  attempt  to  show  that  winning  engagements  is 
totally  worthless  as  an  indicator  of  ACM  skill  level.  The  ability  to 
win  engagements  may  be  the  single  most  important  measure  of  pilot  skill 
level.  The  measure,  however,  can  be  contaminated  by  the  presence  or 
absence  of  many  other  variables  and  these  potential  confounding  variables 
must  also  be  examined  in  order  to  obtain  a  more  valid  measure  of  the 
pilot's  ACM  performance. 

Energy  Management.  The  pilot  who  manages  his  energy  well  probably 
has  a  significant  advantage  over  the  pilot  who  manages  his  energy  less 
well.  A  recent  factor-analytic  study  (Deberg,  1977)  confirmed  that 
energy  measures  provide  the  most  important  single  measurement  class  to 
be  used  in  empirical  descriptions  of  an  air  combat  engagement.  The 
study  did  not  attempt  to  show,  however,  how  the  outcomes  of  engagements 
are  influenced  by  judicious  energy  management. 

Aggressiveness  .  The  trait  most  frequently  cited  by  the  experts 
during  interviews  was  aggressiveness.  The  aggresive  pilot 
spends  little  time  in  a  neutral  situation.  He  acts  boldly  making  his 
opponent  react  to  his  moves.  He  is  eager  to  get  into  the  fight  quickly 
and  may,  therefore,  tend  to  employ  lead  pursuit,  rather  than  pure  or  lag 
pursuit,  when  closing  on  his  adversary.  Some  of  the  experts  predicted 
a  strong  correlation  between  aggressiveness  and  the  use  of  the  roll  axis 
of  the  aircraft.  An  important  weapon  of  the  aggressive  pilot  is  intimi¬ 
dation.  He  makes  feinting  moves  often  and  will  occasionally  point  his 
aircraft  nose  at  the  opponent  for  a  high  angle  gunshot,  more  for  the 
psychological  effect  than  as  a  serious  kill  attempt. 

The  aggressive  pilot  probably  makes  more  use  of  the  vertical  dimen¬ 
sion  for  maneuvering  than  those  who  are  less  aggressive.  This  use  of 
the  vertical  results  in  better  control  of  energy  states  than  is  possible 
for  a  pilot  who  is  maneuvering  strictly  in  the  horizontal,  as  well  as 
providing  another  dimension  to  the  information  the  opponent  must  consider. 

Situation  Awareness.  The  highly  skilled  ACM  pilot  maintains  a  keen 
awareness  of  the  changing  situation  around  him.  He  knows  where  he  is  in 
relation  to  the  terrain,  friendly  forces  and  opposing  forces.  He  is 
aware  of  where  his  opponent  is  and  what  he  is  doing.  He  knows  the 
relative  velocities  and  velocity  rates  of  his  aircraft  and  that  of  his 
opponent.  He  is  able  to  extrapolate  his  position  and  that  of  his 


opponent  into  the  future  in  three  dimensions.  From  this  extrapolation, 
he  knows  when  he  is  about  to  enter  the  effective  envelope  of  one  or 
another  of  his  weapons  and  already  has  that  weapon  selected  when  he 
enters  the  envelope.  Finally,  the  experienced,  skilled  ACM  pilot 
develops  what  was  described  as  a  "sixth  sense"  concerning  his  fuel 
status.  Rather  than  having  to  divert  his  attention  into  the  cockpit 
to  constantly  monitor  his  fuel  supply,  he  is  almost  instinctively  aware 
of  approaching  "bingo"  fuel. 

Situation  awareness  may  be  one  of  the  most  difficult  characteristics 
to  measure  empirically.  It  is  only  readily  noticed  during  moments  when 
it  fails  and  an  aircraft  hits  the  ground,  turns  in  the  wrong  direction, 
runs  out  of  fuel,  or  commits  some  similar  blunder.  Good  situation 
awareness  is  reflected  by  an  absence  of  these  types  of  errors.  It  is 
probable  that  one  element  of  situation  awareness  is  the  ability  to 
efficiently  timeshare  attention  between  the  many  aspects  involved  in  ACM. 

Knowledge  of  ACM.  The  winning  ACM  pilot  spends  a  large  amount  of 
his  ground  time  studying  and  thinking  about  ACM.  He  knows  all  the 
available  information  about  his  aircraft,  weapons  and  tactics  and  about 
the  opponent  aircraft,  weapons  and  tactics.  He  is  especially  aware  of 
the  effective  envelopes  of  his  weapons.  Because  of  this  knowledge,  he 
fires  a  larger  percentage  of  ordnance  with  a  high  probability  of  kill. 

Piloting  Skill .  Much  useful  information  about  a  pilot's  level  of 
ACM  skill  can  be  found  by  examining  the  way  he  flys  the  aircraft.  The 
highly  skilled  ACM  pilot  is  able  to  fly  the  aircraft  to  the  limits  of 
its  performance  parameters  when  necessary.  It  is  important  to  note  that 
this  does  notjnean  that  he  constantly  maneuvers. the  aircraft  at  the 
maximum  angle  of  attack,  but  that  he  is  able  to  maximize  the  tradeoff 
involved  between  performance  and  energy. 

Probably  sane  information  can  be  obtained  by  examining  the  pilot's 
control  inputs.  First,  the  highly  skilled  pilot  is  probably  more 
active  in  his  thrust  control  than  is  the  less  skilled  pilot.  The 
skilled  pilot  is  constantly  shifting  from  idle  power,  to  MIL  power,  to 
afterburner.  The  less  skilled  pilot  is  more  likely  to  remain  at  a 
relatively  high  power  setting  throughout  an  engagement. 

There  was  some  disagreement  among  the  experts  about  whether  control 
smoothness  provides  an  indication  of  ACM  skill.  There  was  consensus 
that  control  smoothness,  especially  aileron  control,  is  essential  at 
high  AOA  to  prevent  loss  of  control.  In  low  AOA  regimes  the  importance 
of  smoothness  is  less  clear  and  this  question  warrants  investigation. 
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Application  of  BFM.  BFM  is  used  largely  to  maneuver  into  weapons 
parameters  against  a  non-maneuvering  target.  BFM,  however,  forms  the 
foundation  on  which  the  maneuvers  during  an  air  combat  engagement  are 
based.  The  highly  skilled  ACM  pilot  is  able  to  effectively  apply  BFM, 
using  segments  of  several  maneuvers,  in  combination  or  in  smooth 
succession,  in  order  to  achieve  a  valid  ordnance  delivery.  One 
important  rule  of  thumb  in  successful  application  of  BFM  is  that  the 
attacker  should  never  point  his  aircraft's  nose  at  the  foe  until  within 
weapons  parameters  and  ready  to  fire.  A  common  tendency  among  less 
skilled  and  experienced  ACM  pilots  is  to  point  the  aircraft  at  the 
opponent  early  in  the  engagement  and  to  attempt  to  decrease  aspect 
angle  and  angle-off  while  still  pointing  at  the  opponent. 

Existing  Measurement  Models 

Three  different  classes  of  measurement  models  currently  in  use  were 
examined,  (a)  the  Automated  Maneuvering  Logic,  (b)  the  Good  Stick  Index, 
and  (c)  the  Air  Combat  Maneuvering  Range  or  Air  Combat  Maneuvering 
Instrumentation. 

Automated  Maneuvering  Logic  (AML).  This  is  a  controller  type  of 
model  which  has  been  described  by  Hankins  (1975) and  by  Burgin,  Fogel  and 
Phelps  (1975).  The  AML  has  been  implemented  on  the  Differential  Maneuver¬ 
ing  Simulator  at  the  NASA  Langley  Research  Center  and  on  the  SAAC  where  it 
allows  a  computer  to  fly  one  cockpit,  simulating  a  reasonably  skilled 
fighter  pilot,  against  a  human  opponent  flying  the  other  cockpit. 

The  actions  of  the  AML  are  based  on  12  decision  rules  concerning 
variables  such  as  the  basic  position  of  the  two  aircraft,  the  visual 
contact  potential  of  the  two  aircraft,  the  line  of  sight  angle  of  the 
two  aircraft,  the  firing  position,  the  closure  rate  and  the  energy 
level.  These  decision  variables  are  then  supplemented  by  information  on 
ground  avoidance  to  prevent  the  AML  from  flight  into  terrain  in  mid¬ 
maneuver. 

The  decisions  of  the  AML  are  based  on  a  5  second  look-ahead.  The 
computer  continuously  predicts  the  effect,  5  seconds  in  advance,  of  each 
of  many  possible  courses  of  action.  The  action  that  will  provide  the 
best  offensive  advantage  at  the  5  second  point  in  the  future  is  then 
undertaken.  The  AML  may  not  duplicate  precisely  the  actions  of  a  highly 
skilled  ACM  pilot.  There  is  little  gamesmanship  and  strategy  involved 
in  its  actions.  It  simply  takes  the  action  most  likely  to  bring  it 
behind  the  other  aircraft  and  into  firing  position.  While  this  is  not 
necessarily  the  optimum  strategy,  the  AML  does  tend  to  win  more 
engagements  than  it  loses. 
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No  detailed  performance  analysis  of  the  AML  has  been  attempted. 
Evaluations  of  AML  engagements  have  been  on  the  engagement  level.  These 
include  such  measures  as  number  of  kills,  number  of  times  killed,  and 
tne  percentage  of  the  engagement  flown  with  an  advantage. 

Good  Stick  Index  (GSI).  This  more  empirically  oriented  measurement 
model  was  developed  by  Vought  Corporation  to  evaluate  the  TAC  ACES  I 
training  program  (USAF,  1977).  During  performance  assessment  using 
the  GSI,  the  pilot  being  tested  flew  attacks  against  a  target  flown  in 
a  predetermined  pattern  by  an  IP.  One  of  the  attacks  flown  for  the 
record  was  a  gun-tracking  exercise  in  which  the  attacker  was  positioned 
behind  the  target  and  was  required  to  close  to  gun  range  and  then  get 
as  many  gun  hits  as  possible  while  the  target  maneuvered.  In  the  other 
kind  of  attack  flown  for  the  record,  the  aircraft  were  set  up  head-on 
and  the  attacker  was  required  to  kill  the  target  as  quickly  as  possible, 
using  any  of  the  weapons  at  his  disposal.  Again,  the  target  aircraft 
flew  a  standard  profile. 

The  GSI  scoring  was  based  on  four  major  measurements,  (a)  tracking 
error,  (b)  time  in  the  pointing  angle  envelope,  (c)  time  to  *irst  kill, 
and  (d)  offensive/defensive  time.  Tracking  error  and  time  -\i  the 
pointing  envelope  were  scored  on  the  gun-tracking  exercise  while  time 
to  first  kill  and  offensive/defensive  time  were  scored  on  the  attacks 
beginning  with  head-on  setups.  Tracking  error  was  defined  as  the 
average  mil  error  when  the  slant  range  was  less  than  3000  feet  with 
the  trigger  depressed.  This  measure  was  used  to  "reflect  the  ability 
to  maintain  a  good  gun-tracking  solution."  Time  in  the  pointing  angle 
envelope  was  defined  in  terms  of  range  and  angular  criteria  and 
"provided  a  measure  of  one's  ability  to  maintain  a  close  gun-tracking 
solution."  Time  to  first  kill  was  the  elapsed  time  between  the  start 
of  a  head-on  engagement  and  the  time  a  kill  was  recorded.  This  measure 
was  selected  because  it  "represented  the  ability  of  the  student  to 
maneuver  in  the  most  expeditious  manner  to  a  firing  envelope  and 
correctly  launch  a  weapon."  Finally,  offensive/defensive  time  was  the 
difference  between  the  offensive  and  defensive  time  divided  by  the 
time  of  engagement  (180  seconds).  This  parameter  was  used  to  provide 
credit  for  maneuvering  that  allowed  the  attacker  to  threaten  the  target 
and  to  penalize  for  allowing  the  target  to  gain  an  advantage.  These 
data  were  then  arbitrarily  weighted  and  summed  in  order  to  provide  a 
single  total  pilot  score  between  0  and  1000  points. 

Several  problems  are  inherent  in  the  way  the  GSI  is  currently  used. 
First,  the  four  raw  scores  are  weighted  according  to  expert  opinion. 

There  is  no  empirical  verification  fiat  the  weights  chosen  are  the 
optimal  value.  Second,  there  is  a  relatively  large  amount  of  unexplained 
variability  in  the  GSI.  A  follow-up  project  is  currently  in  progress 
at  Vought  which  is  exploring  these  potential  problems  (Moore,  Madison, 
Sepp,  Stracener  and  Coward,  1979). 


Finally,  the  application  of  the  GSI  currently  provides  no  diagnostic 
information  to  the  instructor.  A  single  score  is  given  at  the  end  of  a 
set  of  sorties  which  may  give  an  indication  of  overall  ACM  performance. 

Air  Combat  Maneuvering  Range  or  Air  Combat  Maneuvering  Instrumenta¬ 
tion  (ACMR/ACMI).  These  systems  have  not  been  used  to  the'i.'  full 
potential  for  training.  In  the  strictest  sense,  the  ACMR/ACMI  is  a 
simulator  of  weapons  delivery.  The  system  electronically  tracks  up  to 
eight  aircraft  and,  through  telemetry,  is  able  to  compute  precisely  all 
interrelationships  between  these  aircraft.  When  one  combatant  launcnes 
a  simulated  missile  or  gunshot,  the  ground-based  computers  calculate 
the  path  of  the  ordnance  and,  after  adding  a  factor  for  the  potential 
unreliability  of  the  weapon  system,  indicates  whether  the  target  air¬ 
craft  was  killed. 

To  make  these  calculations,  the  ACMR/ACMI  must  collect  and  store  huge 
amounts  of  data  concerning  aircraft  position,  states  and  switchology. 

This  data  pool  has  been  used  for  little  other  than  to  provide  a  playback 
capability  during  debriefing.  Recently,  however,  some  attempt  has  been 
made  to  more  fully  exploit  the  ACMPM  capability  of  the  ACMR/ACMI 
(Simpson,  1976;  Simpson  &  Oberle,  1977).  This  effort  has  produced  the 
software  for  the  calculation  of  an  ACM  Performance  Index  (PI)  based  on 
aircraft  relative  positions  and  position  rates,  aircraft  capabilities, 
and  various  energy  related  measures.  This  approach  has  shown  consider¬ 
able  promise  during  preliminary  testing  but  no  formal  empirical  vali¬ 
dation  of  its  ability  to  discriminate  between  pilots  of  different  skill 
levels  has  been  completed.  Further,  the  algorithm  does  not  account  for 
several  potentially  important  variables  describing  the  way  the  aircraft 
is  flown,  for  example,  control  activity. 

The  ACMR/ACMI,  because  of  Its  data  handling  and  storage  capability, 
can  provide  the  basis  for  an  excellent  ACMPM  system.  With  the  refine¬ 
ment  of  ACMPM  technology,  the  ACMR/ACMI  could  be  used  for  the  assessment 
of  pilot  performance  as  well  as  being  the  effective  testbed  for  aircraft 
and  tactics  it  now  represents.  ACMPM  technology,  however,  still  lags 
far  behind  the  available  hardware  technology. 

The  Technical  Challenge 

Several  aspects  of  the  ACM  task  make  it  different  from  other  in¬ 
flight  tasks  and  make  pi lo~.  performance  measurement  a  real  challenge. 

No  Profile.  Most  previous  empirical  performance  measurement  work 
has  been  conducted  for  instrument  flight  or  specific  visual  maneuvers 
in  which  the  desired  profile  of  the  aircraft  was  well  known  and  for 
which  performance  criteria  of  some  sort  have  been  established.  In  ACM 
there  are  no  fixed  profiles  against  which  to  measure  error,  the  refer¬ 
ence  datum  is  constantly  changing,  and  performance  criteria  are  vague. 
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Existing  models  such  as  Automated  Maneuvering  Logic  (AML)  offer  the 
possibility  of  a  profile  of  sorts,  or  at  least  a  logical  decision  net¬ 
work,  but  no  claims  are  made  that  the  models  are  either  representative 
of  what  a  pilot  does,  or  an  optimum  solution.  They  also  disregard  what 
may  be  important  elements  of  the  job  such  as  gamemanship,  intimidation 
and  faking. 

ACM  Is  Reactive.  The  dynamic  relationship  between  the  two  aircraft 
in  Ivl  is  constantly  changing  because  each  pilot  conti nously  maneuvers 
to  counteract  the  maneuvering  of  the  other.  The  performance  that 
results  is  a  composite  of  a  pair  of  pilots  and  aircraft  with  mutually 
exclusive  objectives. 

Dissimilar  Aircraft.  The  performance  and  weapon  system  capability 
of  the  opponent  aircraft  are  important  tactical  considerations.  This 
implies  that  pilot  performance  should  be  different  when  maneuvering 
against  different  aircraft.  If  pilot  performance  is  different,  the 
measurement  model  that  results  from  an  empirical  study  should  reveal  the 
nature  of  the  differences.  It  is  important,  therefore,  to  include  at 
least  a  sample  of  dissimilar  aircraft  engagements  during  data  collection 
for  measurement  model  development. 

Pilot  Skill  Level.  In  previous  measurement  work,  the  method  of 
sel ecti ng  and  wei ghti ng  final  measures  of  training  performance  dependec 
on  identifying  two  or  more  groups  of  pilots,  whose  skill  level  was 
beyond  question  or  was  established  by  empirical  measurement  of  their 
performance.  The  same  logic,  to  collect  data  on  pilots  of  different 
skill  levels,  is  important  to  the  development  of  training  performance 
measurement  for  ACM.  Unfortunately,  there  is  no  good  way  to  quantify 
pilot  skill  in  ACM  at  the  present  time.  In  fact,  that  is  why  this  work 
is  so  important. 

Summary 

The  Phase  I  Problem  Analysis  consisted  of  (a)  a  review  of  current  and 
future  training,  (b)  an  analysis  of  the  elements  of  fighter  pilot  per¬ 
formance  which  were  judged  to  be  important  by  experts,  (c)  a  review  of 
existing  models  of  ACM  performance  and  (d)  an  analysis  of  the  factors  of 
the  ACM  task  which  create  a  major  technical  challenge  for  performance 
measurement.  The  results  of  this  work  guided  the  data  collection 
(Phase  2)  and  analysis  (Phase  3)  approaches  which  are  described  in  the 
remainder  of  this  report. 


III.  PHASE  2:  DATA  COLLECTION 


Pi lot/ system  performance  and  rating  form  data  were  collected  while 
pilots  of  different  experience  levels  flew  ACM  against  each  other  on  the 
Simulator  for  Air-to-Air  Combat  ( SAAC ) -  The  equipment  and  procedures 
used  are  described  below  in  terms  of  the  (a)  simulator,  (b)  pilots, 

(c)  general  procedures,  (d)  pretest  procedures,  (e)  free  engagement 
procedures,  and  (f)  raw  data  collected. 

Simulator 


SAAC  is  located  at  Luke  AFB,  Arizona,  and  is  used  for  training  air- 
to-air  combat.  It  has  two  F-4  cockpits  controlled  by  a  common  computer 
system  which  allows  free  air  combat  engagements  between  two  pilots. 

The  visual  system  for  each  cockpit  has  a  296°  horizontal  by  150° 
vertical  field  of  view.  The  visual  scene  is  produced  by  a  combination 
of  a  background  scene  and  aircraft  camera-model  superimposition.  The 
scene  generated  is  displayed  on  Cathode  Ray  Tubes  (CRT's)  viewed  through 
optical  glass  windows  which  provide  continuity  of  the  full  visual  field 
focused  at  infinity.  The  viewed  area  is  monochromatic  with  a  greenish 
cast.  An  analog  system  provides  a  representation  of  the  earth,  horizon, 
and  sky.  The  earth  representation  consists  of  squares,  1/2  nautical  mile 
on  a  side,  displayed  in  four  shades  of  gray.  Each  adjacent  square  is  a 
different  shade.  Horizon  appears  as  the  top  of  a  simulated  haze  band 
between  the  earth  and  the  sky.  The  sky  is  homogenous  except  for  a  sun 
image.  While  the  brightness  of  the  sun  is  not  simulated,  i.e.,  a  pilot 
cannot  hide  in  the  sun,  it  can  be  used  effectively  for  defense  against 
heat-seeking  missiles. 

Two  camera-model  systems  provide  each  pilot  with  a  display  of  the 
opposing  aircraft.  Each  model  is  approximately  .3  m.  in  length  mounted 
on  a  gimbal.  The  gimbal ,  under  control  of  the  computer  system,  moves  the 
model  relative  to  a  television  camera  so  that  the  model  is  viewed  by  the 
opposing  cockpit  in  the  air.  The  aircraft  image  is  electronically  super¬ 
imposed  on  the  visual  scene  at  a  higher  brightness  than  the  background. 
The  pilot  sees  the  target  aircraft  superimposed  over  the  earth,  sky,  hor¬ 
izon  and  moving  relative  to  the  background. 

Each  cockpit  also  has  a  g-seat  system  which  contains  31  inflatable 
bladders  under  computer  control.  Inflation  and  deflation  of  the  bladders 
provides  sustained  acceleration  cues  to  the  pilot.  Cues  to  sustained 
g-forces  are  provided  through  a  g-suit  system  identical  to  those  used  in 
actual  aircraft.  The  g-suit  is  inflated  and  deflated  by  a  compressed 
air  system  also  controlled  by  computer  command. 
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Although  each  cockpit  is  mounted  on  a  six-degree-of-freedom  syner¬ 
gistic  motion  system,  motion  was  not  used  for  this  study.  In  the  judge¬ 
ment  of  the  current  users  of  SAAC,  the  visual,  g-seat  and  g-suit  systems 
produce  sufficiently  vivid  motion  cues  to  make  operation  of  the  platform 
motion  systems  unnecessary. 

SAAC  is  controlled  by  a  Xerox  SIGMA  5  Computer  System  with  three 
central  processing  units.  An  operator/instructor  controls  the  simulator 
through  a  corsole  equipped  with  an  Adage  graphic  display  system  which 
portrays  a  three-dimensional  view  of  the  two  simulated  aircraft.  In  addi¬ 
tion  to  the  F-4  characteristics,  one  of  the  cockpits  can  be  operated  with 
flight  characteristics  similar  to  the  MIG-21  aircraft.  Operation  of  the 
radar  system,  which  in  the  actual  F-4  is  handled  by  a  Weapons  System  Offi¬ 
cer  (K50),  is  controlled  by  the  computer  system.  Ail  aircraft  flight 
systems  and  weapons  controls  available  in  an  F-4  are  included  and  operate 
in  each  cf  the  simulator  cockpits. 

Pilots 

i-ty  Air  Force  pilots,  all  F-4  qualified,  participated  in  the  study. 
A  mi  purpose  of  the  study  was  to  develop  measures  which  are  sensitive 
to  umerences  in  air  combat  skill  level.  Accordingly,  the  pilots  were 
selected  for  assignment  to  one  of  three  proficiency  groups. 

Pilots  assigned  to  Group  1  were  the  pilots  with  the  lowest  recent 
experience  in  ACM.  Pilots  in  this  group  included  students  who  had  just 
comoleted  the  ACM  phase  of  the  Replacement  Training  Unit  (RTU)  course. 

It  also  included  some  more  experienced  pilots  who  had  been  assigned  to 
ground  attack  units  and  had  little  or  no  recent  ACM  training. 

Pilots  assigned  to  Group  2  were  currently  undergoing  ACM  training  in 
the  TAC  ACES  program,  an  intensive  course  in  air-to-air  combat.  The 
pilots  in  the  group  typically  were  recent  RTU  graduates.  Most  pilot 
skill  differences  between  Group  1  and  Group  2  can  be  attributed  to  the 
training  received  in  the  TAC  ACES  program 

Pilots  assigned  to  Group  3  were  those  with  the  highest  skill  and 
most  experience  in  ACM.  The  group  consisted  largely  of  TAC  ACES  instruc¬ 
tor  pilots  and  all  pilots  in  the  group  had  considerable  ACM  experience  in 
the  F-4  aircraft  arid  in  the  SAAC  simulator. 

Measures  of  the  training  experience  levels  of  the  pilots  in  the 
three  groups  are  shown  in  Table  1. 


TABLE  1 .  PILOT  EXPERIENCE  DATA 


Skill  Level  Group  . 

Experience  Factors 

1 

2 

3 

Median: 

Age 

30 

24 

29 

Total  Flight  Hours 

1040 

325 

1675 

F-4  Hours 

672 

90 

1375 

Fighter  Hours 

672 

90 

1500 

Flight  Hours  Last  6  Months 

75 

90 

120 

Total  ACM  Engagements 

35 

23 

275 

Air-to-Air  Simulator  Engagements 

5 

10 

60 

Engagements  Last  6  Months 
(Simulator  and  Actual  Aircraft) 

12 

23 

50 

Number: 

Attended  TAC  ACES  Courses 

2 

10 

7 

General  Procedures 


ACM  data  were  collected  3  successive  days  a  week  for  5  consecutive 
weeks.  Six  different  pilots  participated  each  week.  Procedures  and 
conditions  for  data  collection  were  identical  during  each  of  the  5  weeks. 
Table  2  summarizes  the  data  collection. 

On  Day  1,  the  pilots  were  briefed  on  the  background  and  purpose  of 
the  study.  Those  pilots  who  had  not  previously  flown  the  SAAC  were 
given  an  audio-visual  orientation  and  briefing  on  the  SAAC.  The  briefing 
was  followed  by  approximately  30  minutes  of  familiarization  flight  in¬ 
struction  in  the  SAAC  by  one  of  the  project  pilots.  Day  1  activities 
were  concluded  by  5  successive  pretest  engagements  which  are  described 
later. 
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TABLE  2.  SUMMARY  OF  DATA  COLLECTION 
Lapsed  time  of  data  collection 
Approximate  duration  of  each  run 
Number  of  pilots  in  each  group 
Total  number  of  pilots 

Number  of  pilots  used  each  week  (2  from  each  group) 
Number  of  pretest  engagements 
Number  of  lvl  engagements  F-4  configuration 
Number  of  lvl  engagements  F-4/MI6-21  configuration 


5  weeks 
3.5  minutes 


30  per  week 
45  per  week 
6  per  week 


On  Day  2  and  Day  3  the  pilots  flew  round  robin  ACM  engagements. 

Three  successive  engagements  were  flown  in  each  sortie,  with  a  1  or  2 
minute  break  between  engagements  while  the  simulator  was  reset.  Each 
pilot  flew  2  sorties  on  one  day  and  3  sorties  on  the  other.  With  few 
exceptions,  the  pilots  had  a  20  to  40  minute  break  between  each  sortie. 
Except  for  different  setup  conditions  and  schedules,  the  data  collection 
procedures  were  generally  the  same  for  all  three  days. 

One  of  the  two  project  pilots  operated  the  console,  communicated 
with  subject  pilots  in  the  cockpits  and  additional  project  personnel 
performed  manual  data  recording.  After  the  subject  pilots  were  in  the 
cockpits  and  ready,  the  project  pilot  read  the  instructions  for  each 
engagement  to  the  pilots,  checked  that  the  initial  setup  conditions  (which 
were  preprogrammed )were  correct  and  started  the  engagement  by  pressing  a 
control  button  on  the  console.  While  the  instructions  were  being  read, 
one  of  the  project  personnel  entered  the  engagement  number  on  a  set  of 
thumbwheels  to  mark  the  data  tape  and  started  the  data  recording  on  the 
nine- track  magnetic  tape  when  the  engagement  began. 

Each  engagement  ran  for  a  maximum  of  approximately  3.5  minutes. 

The  maneuvering  could  be  observed  in  real  time  on  a  graphic  display  which 
portrayed  the  actions  of  the  aircraft  in  three  dimensional  perspective. 
The  display  also  contained  a  number  of  relevant  variables  such  as  air¬ 
speed,  range  between  the  aircraft  and  altitude.  One  of  the  project 
personnel  made  notes  on  events  such  as  restarting  an  engagement,  over-g 
(exceeding  the  g  limits  of  the  aircraft)  and  the  outcome  of  the  engage¬ 
ment.  At  the  end  of  each  engagement,  conditions  were  reset  for  the  next 
engagement  and  the  procedure  was  repeated  until  the  end  of  the  engagement 
sequence. 


At  the  end  of  Day  2  and  Day  3,  the  project  pilots,  based  on  their 
observations  of  the  console  display,  filled  out  a  rating  form  for  each 
of  the  subject  pilots.  At  the  end  of  Day  3,  the  project  pilots  also 
rank  ordered  the  6  subject  pilots  in  terms  of  their  demonstrated  ACM 
skill.  Each  of  the  subject  pilots  filled  out  an  identical  ranking,  in¬ 
cluding  themselves,  based  on  their  experience  of  flying  against  each  of 
the  other  pilots. 

Pretest  Procedures 

The  purpose  of  the  pretest  was  to  document  the  entry  skill  level  of 
the  pilots  while  flying  against  a  maneuvering,  but  non-interactive  target. 
Each  pilot  flew  five  times  against  a  target  aircraft  which  followed  a 
predetermined  flight  profile  on  instruments  without  regard  to  the  man¬ 
euvering  of  the  attacking  aircraft.  The  five  engagements  were  subsumed 
under  three  maneuvers  which  had  different  initial  conditions  and  objec¬ 
tives.  Maneuvers  I  and  II  involved  only  one  engagement  each.  Maneuver 
III  involved  three  engagements  with  the  same  initial  conditions  but  dif¬ 
ferent  maneuvering  by  the  target  aircraft.  For  all  maneuvers,  one  of  the 
project  pilots  flew  the  target  aircraft  through  the  specified  profiles. 
Subject  pilots  were  not  allowed  to  observe  the  console  display  while 
waiting  for  their  scheduled  engagements. 

Maneuver  I.  The  purpose  of  Maneuver  I  was  to  obtain  data  on  the 
ability  of  the  pilot  to  hit  the  target  aircraft  with  20  mm  cannon  fire 
during  a  brief  engagement.  The  engagement  started  with  the  attacking 
aircraft  a  short  distance  behind  the  target  aircraft.  Specific  initial 
conditions  are  shown  in  Table  A1  (in  Appendix  A).  The  instructions  to 
the  attacking  pilot  and  the  target  pilot  are  shown  in  Table  A2.  The 
maneuver  ended  when  the  target  pilot  called  "cease  maneuver." 

Normally,  the  SAAC  stops  when  an  aircraft  is  "killed"  by  gun  or 
missile  fire.  This  can  be  prevented,  however,  by  use  of  a  KILL  OVERRIDE 
feature  which  counts  cumulative  hits  but  does  not  stop  the  simulation. 

KILL  OVERRIDE  was  on  during  Maneuver  I  to  allow  tallying  of  the  total 
number  of  20  mm  cannon  hits  on  the  target  aircraft. 

Maneuver  II.  The  purpose  of  Maneuver  II  was  to  obtain  data  on  the 
ability  of  the  pilot  to  hit  the  target  aircraft  with  both  heat-seeking 
missiles  (AIM-9J)  and  20  mm  cannon  fire.  The  engagement  started  with  the 
attacking  aircraft  well  behind  the  target  aircraft.  The  target  remained 
straight  and  level  until  informed  by  the  console  operator  that  the  attack¬ 
er  had  closed  to  the  6,000  foot  range.  The  target  then  began  to  maneuver. 
The  specific  initial  conditions  for  this  maneuver  are  shown  in  Table  A3. 
The  instructions  to  the  attacking  pilot  and  the  target  pilot  are  shown  in 
Table  A4.  The  maneuver  ended  after  the  attacking  aircraft  had  passed 
the  target  aircraft  and  the  separation  between  them  exceeded  6,000  feet. 
KILL  OVERRIDE  was  on  during  this  maneuver. 
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Maneuver  III.  The  purpose  of  Maneuver  III  was  to  obtain  data  on 
the  pilot's  ability  to  expeditiously  kill  the  target  aircraft  using 
heatseeking  missiles  or  20  tin  cannon  fire.  Three  engagements,  with  the 
same  starting  conditions,  were  flown.  The  engagement  began  with  the 
attacking  and  target  aircraft  passing  head-on.  The  target  aircraft  then 
maneuvered  according  to  the  prescribed  profile  for  the  engagement.  The 
exact  initial  conditions  for  this  maneuver  are  shown  in  Table  A5.  In¬ 
structions  given  to  the  attacking  pilot  and  the  target  pilot  are  shown 
in  Table  A6.  KILL  OVERRIDE  was  off  for  these  three  engagements.  The 
engagement  ended  when  a  kill  occurred  or  bingo  fuel  (i.e.,  minimum  allow¬ 
able  fuel  remaining)  was  reached.  If  a  kill  did  not  occur,  the  length 
of  an  engagement  was  approximately  3.5  minutes. 

Free  Engagement  Procedures 

A  round  robin  of  free  engagements  was  flown  among  the  six  pilots 
participating  during  each  week.  Every  pilot  fought  each  of  the  other 
five  pilots  in  three  successive  engagements.  This  round  robin  took 
place  over  the  second  and  third  day  of  data  collection.  A  pilot  usually 
fought  two  other  pilots  on  one  day  and  the  remaining  three  pilots  on 
the  following  day,  or  vice  versa. 

In  addition,  on  the  third  day  the  two  pilots  considered  to  be  the 
most  experienced  flew  two  additional  sorties  of  three  engagements  in  which 
one  of  the  aircraft  was  programmed  to  have  the  same  flight  characteristics 
as  a  MIG-21.  Each  of  these  two  pilots  would  fly  one  sortie  in  the  F-4 
configuration  and  one  in  the  MIG  configuration.  These  pilots  all  had 
prior  experience  with  MIG-21  dynamics  on  the  SAAC. 

Different  initial  conditions  were  used  for  each  engagement  in  the 
sortie  of  three.  The  same  three  sets  of  initial  conditions  were  used, 
however,  for  each  sortie  of  similar  aircraft  engagements  (F-4  vs.  F-4) 
and  the  two  sorties  of  dissimilar  aircraft  engagements  (F-4  vs.  MIG). 

The  only  exception  was  that  the  weapons  available  for  the  dissimilar  air¬ 
craft  engagements  were  different  than  those  available  for  the  similar 
aircraft  engagements.  Tables  A7  to  A9  show  the  initial  conditions  of 
both  the  similar  and  dissimilar  aircraft  engagements.  The  object  of  each 
engagement  was  to  kill  the  other  aircraft. 

For  the  similar  aircraft  engagements  each  aircraft  had  a  full  load, 
640  rounds, of  20  mm  cannon  ammunition  and  two  heat  seeking  missiles  (AIM- 
9J).  Only  heat  missiles  and  no  radar  missiles  were  provided  in  order  to 
force  the  pilots  to  engage  in  close  maneuvering.  Radar  missiles  were 
allowed  for  the  F-4,  however,  for  dissimilar  aircraft  engagements  because 
of  the  maneuvering  advantage  of  the  MIG-21. 

Bingo  fuel  limits  were  set  to  allow  engagements  to  run  approximately 
3.5  minutes  unless  a  kill  occurred  first.  A  few  rules  were  established 
for  the  engagements  which  either  furthered  the  purpose  of  the  study  or 
overcame  peculiarities  of  the  SAAC.  These  rules  were: 
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1.  Upon  reaching  bingo  fuel,  the  pilot  had  to  call  it  out  to  the 
console  operator,  separate  from  the  engagement  and  head  in 
the  "home"  direction,  southerly  for  one  aircraft  and  northerly 
for  the  other.  If  the  pilot  failed  to  call  bingo  fuel  and 
separate,  a  kill  was  credited  to  the  other  pilot.  The  engage¬ 
ment  did  not  end  until  a  kill  occurred  or  both  aircraft  reached 
bingo  fuel . 

2.  Beam  and  head-on  heat  missile  kills  were  not  permitted  although 
the  SAAC  would  allow  them.  That  is,  a  kill  was  not  allowed  if 
an  aircraft  had  an  aspect  angle  90°  or  greater  from  the  tail 
and  a  missile  was  fired  at  him.  In  these  cases  the  console 
operator  would  momentarily  invoke  the  KILL  OVERRIDE  function 
which  would  prevent  the  missile  from  killing  th®  aircraft. 

3.  6-forces  were  an  important  consideration  in  the  engagements. 

At  between  6  and  6.5  g's,  the  visual  display  would  darken  to 
simulate  blackout.  If  the  pilot  pulled  8.5  g's  a  warning 
buzzer  would  sound  and  the  console  operator  reminded  the  pilot 
he  had  over-g'd  but  the  engagement  would  continue.  If  a  pilot 
pulled  9.5  g's  or  more,  he  would  have  over-stressed  an  actual 
aircraft  and  therefore  a  kill  was  awarded  to  the  other  pilot 
under  these  conditions. 

4.  If  a  pilot  flew  into  the  ground  during  the  engagement,  a  kill 
was  awarded  to  the  other  aircraft.  If  both  aircraft  flew 
into  the  ground,  a  dual  kill  was  declared. 

5.  To  achieve  a  kill  with  the  20  mm  cannons  a  total  of  3  hits  had 
to  strike  the  other  aircraft.  Hits  on  non-critical  aircraft 
surfaces  were  scored  as  fractions  of  a  hit. 


6.  If,  during  the  dissimilar  aircraft  engagements,  the  2  aircraft 
separated  by  5  miles  or  more,  they  were  required  to  close  to  a 
distance  of  3  miles  before  a  weapon  could  be  fired.  This  rule 
prevented  the  F-4  pilot  from  opening  to  a  large  distance  and 
firing  a  radar  missile  during  the  dissimilar  aircraft  engagements. 

Raw  Data  Collected 

Four  kinds  of  data  were  collected  during  the  study.  There  were: 

(a)  continuous  samples  of  aircraft  system  variables;  (b)  a  computer 
printout  of  selected  variables;  (c)  engagement  outcomes  and  events; 
and  (d)  evaluations  of  ACM  performance  in  the  form  of  pilot  ratings 
and  rankings. 


Aircraft  Variables.  The  most  comprehensive  data  were  taken  by 
automatic  recording  of  67  aircraft  variables  from  each  aircraft.  The 
basic  data  sets  included  aircraft  position,  altitude,  airspeed,  orienta¬ 
tion,  angle  of  attack,  g-forces,  fuel  flow,  control  positions  and  force 
applied,  aircraft  control  and  armament  switch  settings,  radar  functions 
employed,  weapons  fired  and  hits.  Table  C2  is  a  complete  listing  of  the 
variables  recorded.  All  variables  were  sampled  twice  per  second  (2  Hz) 
except  for  control  positions  and  forces  which  were  sampled  ten  times  per 
second  (10  Hz). 

The  data  were  recorded  on  nine-tract  magnetic  tape  in  digital  form. 
Data  recording  was  started  and  stopped  manually  at  the  beginning  and  end 
of  each  engagement  by  one  of  the  project  personnel. 

Scoring  Print.  The  SAAC  has  the  capability  to  print  a  time-marked 
history  of  major  aircraft  variables  and  events  that  occur  during  an 
engagement  and  summary  data  at  the  end  of  the  engagement,  i.e.,  who  Killed 
whom,  the  kill  weapon,  the  number  of  rounds  of  each  weapon  expended,  and 
the  total  engagement  time.  The  information  available  in  the  Scoring 
Print  is  shown  in  Table  3.  Scoring  Print  information  was  obtained  at 
the  same  time  as  the  tape  recorded  data  described  above.  While  much  of 
the  data  obtained  by  these  two  automatic  methods  is  the  same,  the  Scoring 
Print  provided  a  succinct  summary  of  the  key  aircraft  variable  and  event 
occurrences  during  an  engagement  and  aided  interpretation  of  the  other 
data. 


Engagement  Outcomes  and  Events.  During  the  fixed  profile  pretest 
engagements,  data  on  weapons  fired,  number  of  hits,  and  kills  were 
manually  recorded.  The  fixed  profile  engagements  were  constructed  pri¬ 
marily  to  determine  how  rapidly  and  accurately  a  pilot  could  use  a  20  mm 
cannon  and  heat-missies  to  hit  the  target  aircraft.  The  data  were  re¬ 
corded  by  hand  to  provide  a  quick  summary  of  each  pilot's  performance 
during  the  five  fixed  profile  engagements.  A  copy  of  the  data  recording 
form  is  shown  in  Figure  Al. 

During  the  free  engagements,  outcome  data  were  manually  recorded. 

A  free  engagement  could  end  (a)  in  a  draw  because  bingo  fuel  was  reached, 
(b)  one  aircraft  had  killed  the  other  with  weapons,  or  (c)  one  or  both 
aircraft  were  killed  from  exceeding  g  limits,  hitting  the  ground,  or 
ignoring  bingo  fuel  requirements. 

The  outcome  and  cause  were  recorded  on  a  matrix  form  which,  when 
completed,  summarized  the  outcomes  of  the  three  engagements  between  all 
15  possible  pairs  of  pilots.  These  data  were  used  to  provide  information 
to  each  pilot  on  his  own  performance  and  the  performance  of  the  other 
pilots,  and  also  were  used  for  partitioning  the  automatically  recorded 
data  in  terms  of  engagement  outcome  for  analysis  purposes. 


I.  Header  Information 


A.  Pilots’  names 

B.  Aircraft  type 

C.  Date 

D.  Clock  time 

II.  Time  Marked  Information 

A.  Both  aircraft 

1.  Altitude 

2.  Mach 

3.  g  * 

4.  Line  of  sight  to  other  aircraft 

B.  Between  aircraft 

1 .  Range 

2.  Range  rate 

3.  Track  error 

a.  Azimuth 

b.  Elevation 

C.  Significant  activities  or  status 

1.  Radar  lock-on 

2.  Loss  of  radar  lock-on- 

3.  Weapons  selected 

a.  No  weapons  ready 

b.  AIM-9 

c.  AIM-7 

d.  Guns 

4.  Heat  missile  fired 

5.  Radar  missile  fired 

6 .  Guns  f i  red 

7.  Trigger  pulled  -  no  weapons  ready 

8.  Kill  -  missile 

9.  Kill  -  guns 

10.  No  kill 


TABLE  3.  SCORING  PRINT  INFORMATION  (Cont.) 


a.  Missile  -  (reason  for  miss,  boresight  angle, 
range,  etc.) 

b.  Guns  -  number  of  rounds  fired,  number  of  hits 
III.  Engagement  Summary  Information 

A.  Engagement  time 

1.  Elapsed  seconds:  total 

2.  Elapsed  minutes:  seconds  total 

3.  Time  of  first  kill  -  weapon  (guns,  AIM-7,  AIM-9) 

B.  Weapon  status 

1.  AIM-7:  number  available/number  expended 

2.  AIM-9:  number  available/number  expended 

3.  Guns:  rounds  expended/number  of  hits 


Ratings  and  Rankings.  At  the  end  of  each  day  of  free  engagements, 
the  2  project  pilots,  both  ACM  instructors,  rated  each  of  the  6  pilots  on 
five  factors:  (a)  decisiveness,  (b)  situation  awareness,  (c)  maneuvering, 
(d)  aggressiveness,  and  (e)  overall  ACM  performance.  A  5  point  linear 
scale  (sot  figure  A2)  was  used  for  rating  each  factor.  The  criteria 
definitions  were  the  same  as  those  used  by  ACM  instructors  for  evaluating 
ACM  pilot  students  and  therefore  were  familiar  to  the  project  pilots. 

At  the  end  of  the  last  day  of  free  engagements,  each  of  the  partici¬ 
pating  pilots  rank  ordered  the  six  pilots,  including  himself,  in  terms 
of  demonstrated  overall  ACM  ability.  The  6  pilots  were  told  that  the 
ranking  would  be  strictly  confidential. 

Data  Analysis  Facilities.  The  raw  data  were  duplicated  and  trans¬ 
ported  to  the  Naval  Training  Equipment  Center  for  analysis  using  a 
SIGMA-7  computer.  The  two  principal  reasons  for  using  this  facility  were 
(a)  the  investigators  had  developed  specialized  performance  measurement 
analysis  software  on  prior  research  contracts  and  were  familiar  with  the 
system,  and  (b)  the  principal  statistical  performance  measurement  analyst 
was  resident  in  Orlando,  Florida.  Computer  data  analysis  for  measure¬ 
ment  development  is  a  highly  interactive  task  between  a  computer  and  a 
skilled  analyst.  The  task  requires  substantial  experience  w5th  human/ 
system  performance  data,  simulator  data  collection,  the  natu'»  of  the 
simulated  flight  task,  computer  programming  and  multi-variate  statistics. 
It  was  more  cost  effective  to  bring  the  data  to  the  facility  and  experi¬ 
enced  measurement  analyst  than  to  bring  the  analyst  and  specialized  soft¬ 
ware  to  the  data  and  an  unfamiliar  computer.  The  general  data  flow  is 
shown  in  Figure  C5. 


IV.  PHASE  3:  DATA  ANALYSIS 


The  goal  of  the  data  analysis  was  to  find  the  smallest  comprehensive 
set  of  measures  which  would  discriminate  the  differences  between  RTU 
experience  level  pilots  and  acknowledged  expert  pilots  in  Ivl  free  engage¬ 
ment  ACM.  To  perform  this  measurement  analysis,  it  was  methodologically 
necessary  to  first  determine  that  the  pilots  were  assigned  properly  to 
the  three  experience  level  groups.  After  the  group  assignment  was  con¬ 
firmed,  the  analysis  was  conducted  at  two  levels  of  detail,  (a)  general 
measures  of  performance  (i.e.,  energy  used,  amount  of  time  opponent  out 
of  view,  throttle  use,  average  airspeed,  etc.)  which  could.be  taken  over 
a  whole  engagement  without  complex  subdivision  of  the  engagement  or 
maneuvering,  and  (b)  more  detailed  measures  of  performance  within  certain 
segments  of  the  engagement  that  might  reflect  the  tactics  being  employed 
by  the  pilots.  Accordingly,  this  section  is  subdivided  into  three  major 
subsections,  (a)  skill  level  assessment,  (b)  whole  engagement  performance 
measurement  and  (c)  preliminary  tactics  performance  analysis. 

A.  Skill  Level  Assessment 

The  subject  pilots  were  chosen  from  three  somewhat  loosely  defined 
groups  ranging  from  novices  through  "expert"  ACM  instructors.  This 
assured  that  the  pilots  would  represent  a  broad  spectrum  of  experience 
and  skill  levels  in  ACM.  While  this  grouping  of  pilots  provided  a  pre¬ 
liminary  attempt  at  skill  level  classification,  the  investigators  expected 
a  large  degree  of  overlap  in  ACM  skill  among  the  three  groups.  It  was, 
therefore,  the  plan  to  make  a  final  reassignment  of  pilots  into  skill 
level  groups  at  the  conclusion  of  data  collection  on  the  basis  of  the 
data  collected. 

In  order  to  make  this  final  skill  level  assessment  the  investigators 
collected  several  types  of  skill  level  data.  These  included  performance 
on  the  pretest  in  which  each  pilot  made  five  attacks  on  a  target  maneuver¬ 
ing  in  a  predetermined  pattern.  They  included  the  peer  rankings  and  ex¬ 
pert  ratings  collected  at  the  completion  of  each  week's  exercises.  Final¬ 
ly,  they  included  records  of  the  outcomes  of  the  free  engagements  flown 
during  each  week. 

Pretest  Exercises 


In  the  first  pretest  exercise  (Maneuver  I),  the  pilot  was  required  to 
track  a  maneuvering  target  within  gun  range  and  to  score  as  many  hits  as 
possible  within  the  allotted  time.  Of  the  30  pilots  taking  part  in  the 
study,  only  10  scored  a  gun  hit.  This  exercise,  therefore,  provided  littl 
information  which  could  be  used  to  rank  the  30  pilots  according  to  skill 
level. 


The  second  pretest  exercise  (Maneuver  II)  required  the  pilot  to 
fire  an  AIM-9  missile  at  the  target  and  then  to  close  for  a  high  angle 
gun  pass.  In  this  exercise  15  of  the  30  pilots  scored  a  missile  kill 
and  only  two  scored  a  gun  hit.  The  pilots'  performance  on  the  first 
exercise  and  their  performance  on  the  second  showed  no  significant- 
correlation. 

In  the  third,  fourth  and  fifth  exercises  (Maneuver  III)  the  pilots 
were  required  to  attack  a  maneuvering  target  and  to  achieve  a  kill  as 
expeditiously  as  possible.  The  variable  of  primary  interest  here  was 
the  elapsed  time  required  to  score  a  kill.  Table  B1  shows  the  time  to 
kill  for  each  of  the  30  pilots  for  each  of  the  three  attacks  on  the 
target.  While  performance  during  these  three  attacks  did  not  correlate 
with  performance  during  the  first  two  exercises,  it  did  correlate  signi¬ 
ficantly  with  other  measures  of  pilot  skill  level  including  peer  rankings, 
expert  ratings,  and  performance  during  the  subsequent  free  engagements. 
These  correlations  will  be  discussed  later. 


Ranking  Data 

Table  D1  presents  a  summary  of  the  peer  rankings  and  the  expert  rank¬ 
ings  obtained  for  each  pilot  during  each  of  the  5  weeks.  It  is  interest¬ 
ing  to  note  that  in  each  of  the  5  weeks  the  pilots  in  Group  3  (pilots 
number  5  and  6)  were  ranked  as  the  top  2  pilots  in  both  the  peer  rankings 
and  in  the  project  pilot  rankings.  Pilots  in  Group  2  (pilots  number  3 
and  4)  generally  fell  in  the  middle  of  the  rankings  while  pilots  in  Group 
1  (pilots  number  1  and  2)  fell  at  or  near  the  bottom  of  the  rankings. 

Also  interesting  is  the  strong  degree  of  correspondence  between  the 
expert  rankings  and  the  average  of  the  peer  nnkings.  This  indicates 
that  a  strong  interobserver  reliability  is  possible  when  rating  overall 
ACM  performance  in  the  SAAC. 

Expert  Ratings 

In  addition  to  the  ranking  forms,  the  project  pilots  were  asked  to 
rate  each  of  the  pilots  on  a  0  to  4  scale  on  several  dimensions  of  ACM 
performance.  The  dimensions  rated  were:  (a)  Decisiveness;  (b)  situation 
awareness;  (c)  maneuvering  skill;  (d)  aggressiveness;  and  (e)  overall 
performance.  Table  D2  gives  the  mean  ratings  for  each  of  the  three  groups 
of  pilots  for  these  dimensions.  For  each  measure  except  aggressiveness, 
the  scores  show  a  consistent  increase  across  the  skill  level  groups.  It 
is  not  surprising  that  aggressiveness  does  not  show  a  perfect  correlation 
with  experience  level.  Aggressiveness  is  perhaps  more  a  personal  trait 
than  the  other  measures  which  depend  on  experience,  training,  and  recency. 


As  might  be  expected,  the  overall  performance  ratings  were  highly 
correlated  with  the  other  scores  of  ACM  performance.  Table  D3  presents 
a  matrix  of  the  intercorrelations  among  the  five  dimensions  in  the  expert 
ratings  as  well  as  the  peer  rankings  and  project  pilot  rankings. 

Two  cautions  should  be  emphasized  when  looking  at  this  table.  The 
first  involves  the  negative  correlations  between  the  ranking  data  and 
the  rating  data.  In  the  case  of  the  ranking  data,  a  low  score  indicates 
the  best  performance  while  the  reverse  is  true  for  the  rating  data.  The 
high  negative  correlations,  therefore,  indicate  that  pilots  tended  to 
be  similarly  rated  in  both  types  of  scales.  Secondly,  these  correlations 
may  be  spuriously  low.  Several  statistically  important  assumptions  may 
have  been  violated  in  the  calculation  of  the  correlations  between  the 
rating  data  and  the  ranking  data.  The  ranking  data  only  achieves  ordinal 
level  scaling  while  the  correlational  analysis  requires  that  at  least 
interval  scaled  data  be  used.  In  addition,  the  desired  linear  relation¬ 
ship  between  the  two  sets  of  correlated  data  may  not  exist.  If  so, 
this  would  explain  why,  for  example,  the  correlation  between  the  expert  • 
ranking  and  the  overall  performance  rating  is  -.856  when,  desirably,  it 
should  approach  -1.000. 

Engagement  Outcomes 

The  most  commonly  accepted  measure  of  ACM  skill  level  is  the  number 
of  kills  achieved.  Therefore,  the  outcomes  of  the  free  engagements  flown 
by  the  pilots  were  considered  an  important  source  of  information  in 
assessing  skill  level.  Figure  1  shows  that  engagement  outcomes  reveal 
clear  differences  between  the  three  skill  level  groups. 

In  order  to  determine  the  strength  of  relationship  between  some  of 
these  measures  of  pilot  ACM  skill,  rank  correlations  between  them  were 
calculated.  The  pilots  were  first  ranked  from  1  to  30  on  time  to  kill 
on  the  last  three  pretest  exercises,  on  the  expert’s  overall  performance 
ratings  from  the  free  engagements,  and  on  the  total  number  of  kills  during 
the  free  engagements.  Spearman  Rank  Correlations  were  then  calculated 
between  the  pairs  of  these  measures.  The  correlation  between  number  of 
kills  and  overall  performance  rating  was  .695.  The  correlation  between 
time  to  kill  and  overall  performance  ratings  was  .694.  Finally,  the 
correlation  between  time  to  kill  and  number  of  kills  was  .716. 

The  high  correlation  between  the  number  of  kills  and  the  overall 
performance  rating  was  to  be  expected.  The  final  criterion  for  ACM 
performance  is  the  accomplishment  of  the  kill.  In  the  sample,  the  pilots 
achieving  the  largest  number  of  kills  also  tended,  to  have  the  highest 
overall  rating.  Since  this  correlation  was  not  1.000,  however,  there  is 
evidence  that  the  experts  were  also  rating  the  pilots  on  some  dimension 
or  dimensions  other  than  just  kills. 
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The  correlation  between  the  pretest  time  to  kill  and  the  overall 
performance  rating  during  the  free  engagements  is  Interest 'rig.  It 
indicates  that  the  time  to  kill  measure  is  highly  predictive  of  perform¬ 
ance  during  lvl  free  engagements.  This  is  also  indicated  by  the  similarly 
high  correlation  between  time  to  kill  during  pretest  and  the  number  of 
kills  during  free  engagements.  This  evidence  suggests  that  the  pretest 
time  to  kill  is  a  valid  measure  of  ACM  performance,  at  least  in  the  SAAC 
simulator. 

The  purpose  of  the  skill  level  assessment  was  to  provide  the  basis 
for  a  final  assignment  of  the  pilots  into  skill  level  groups.  Examina¬ 
tion  of  the  data  described  above,  howeve  indicates  that  the  preliminary 
group  assignments  provided  us  three  distinctly  different  skill  levels. 

This  is  especially  apparent  in  the  engagement  outcome  data.  Pilots  in 
Group  3  dominated  the  pilots  in  both  Groups  1  and  2  during  the  free  en¬ 
gagements.  While  there  was  not  so  pronounced  a  difference  between  Groups 
1  and  2  in  total  kills.  Group  1  pilots  did  not  achieve  a  single  weapons 
kill  against  a  Group  2  pilot  during  the  5  weeks  of  engagements. 

This  difference  between  the  three  groups  was  also  apparent  in  the 
ranking  data,  the  expert  rating  data  and,  to  a  lesser  extent,  in  the 
pretest  data.  This  study,  therefore,  concluded  that  the  original  assign¬ 
ment  of  pilots  into  skill  level  groups  provides  adequate  group  differ¬ 
ences.  All  further  analyses  were  based  on  the  original  assignments. 

B.  Whole  Engagement  Performance  Measurement 

The  analysis  of  whole  engagement  performance  measures  was  conducted 
in  four  major  steps,  (a)  the  development  of  candidate  measures  from  the 
time  history  data  tapes  of  all  relevant  aircraft  variables,  (b)  an  analy¬ 
sis  of  the  pretest  exercises  using  the  candidate  measures,  (c)  a  univari¬ 
ate  (one  measure  at  a  time)  analysis  of  free  engagements,  and  (d)  a 
multivariate  analysis  of  free  engagements. 

Development  of  Candidate  Measures 

From  the  67  aircraft  and  system  variables  recorded  on  time  history 
data  tapes  from  each  cockpit,  28  measures  of  whole  engagement  performance 
were  calculated.  Some  of  these  measures  were  taken  directly  from  the 
original  67,  while  others  of  the  candidate  set  were  composite  measures 
formed  by  mathematical  combination  of  two  or  more  variables.  The  formu¬ 
lation  of  candidate  measures  was  guided  by  the  results  of  Phase  1.  A 
Qoscription  of  each .candidate  measure  follows: 


1.  Altitude  Rate:  The  arithmetic  mean  of  the  absolute  values 
of  the  altitude  rates  recorded  for  a  pilot  during  a  given 
engagement. 

2.  Out  of  View:  The  percentage  of  time  the  opponent  was  out  of 
the  pilot's  field  of  view  during  a  given  engagement  as 
calculated  by  the  SAAC. 

3.  Airspeed:  The  arithmetic  mean  of  the  values  of  indicated 
airspeed  recorded  for  a  pilot  during  a  given  engagement. 

4.  Speedbrake:  The  average  speedbrake  deflection  (maximum 
deflection  is  10)  during  an  engagement. 

5.  Fuel  Flow:  The  average  fuel,  in  pounds  per  hour,  used  by 
the  pilot  during  the  engagement. 

6.  Relative  Altitude  Use:  The  pilot's  use  of  the 

vertical  dimension  relative  to  his  opponent's  use,  calculated 
as  the  ratio  of  the  given  pilot's  altitude  standard  deviation 
to  that  of  his  opponent  during  a  single  engagement. 

7.  Energy  Management  Index:  A  composite  representing  the  ability 
of  the  pilot  to  obtain  and  maintain  the  maximum  amount 

of  energy  (kinetic  and  potential)  for  a  given  expenditure 
of  fuel.  It  is  a  function  of  remaining  fuel,  fuel  flow, 
airspeed  and  altitude  integrated  across  the  length  of  the 
engagement  as  shown  in  Table  C3. 

8.  Offensive  Time:  The  percentage  of  time  the  given  pilot's 
opponent  is  positioned  at  a  sight  angle  of  less  than  60°. 

9.  Offensive  with  Advantage:  The  percentage  of  time  the  given 
pilot's  opponent  is  positioned  at  a  line-of-sight  of  lass 
than  60°  and  with  an  aspect  angle  of  less  than  90°. 

10.  Throttle  Idle,  The  percentage  of  time  the  throttle  is  in 
the  idle  position  during  an  engagement. 

11.  Throttle  LO  MIL:  The  percentage  of  time  the  throttle  Js  in 
the  LO  MIL  position  during  an  engagement. 

12.  Throttle  HI  MIL:  The  percentage  of  time  the  throttle  is  in 
the  HI  MIL  position  during  an  engagement. 

13.  Throttle  Afterburner:  The  percentage  of  time  the  throttle 
is  in  the  afterburner  position  during  an  engagement. 


42 


RMS  Heading:  The  root-mean-squared  error  about  the 
average  heading. 

Absolute  Average  Heading:  The  average  heading  held 
by  the  pilot  during  the  engagement. 

Lead  Time:  The  percentage  of  time  the  given  pilot  is 
employing  lead  pursuit,  i.e.,  his  aircraft  nose  is 
pointed  ahead  of  his  opponent's  nose  during  an  engagement. 

Time  in  Range:  The  percentage  of  time  the  pilot's 
opponent  is  positioned  with  a  sight  angle  of  less  than 
60°,  an  angle  of  less  than  90°  and  at  a  range  of  less  than 
2000  feet. 

Roll  Rate:  The  average  .absolute  roll  rate,  in  degrees  per 
second,  used  by  a  pilot  during  an  engagement. 

Roll  Rate  Times  Altitude  Rate:  A  composite  measure  of  the 
maneuvering  rate  used  during  an  engagement.  The  roll  rate 
is  multiplied  by  the  altitude  rate  and  the  absolute  value 
is  taken.  This  measure  is  the  mean  value  of  this  product 
during  an  engagement. 

Plane  of  Action:  The  plane  in  which  the  aircraft  is  moving 
at  a  given  instant.  It  is  a  composite  function  of 
X,  Y,  and  Z.  The  measure  is  the  arithmetic  mean  of 
values  recorded  during  an  engagement. 

Defensive  Time:  The  percentage  of  time  that  the  given 
pilot's  opponent  is  positioned  at  a  sight  angle  of  greater 
than  120°,  an  aspect  angle  greater  than  90°  and  at  a  range 
of  less  than  4000  feet. 

Angle  of  Attack:  The  percentage  of  time  that  a  pilot's 
angle  attack  exceeds  28  units  during  an  engagement. 

AIM-9  Success:  The  probability  that  the  pilot  achieved  an 
AIM-9  kill  during  the  given  engagement.  On  the  record, 
an  AIM-9  kill  was  scored  as  a  1  while  any  other  outcome  was 
scored  as  an  0.  The  measure,  therefore,  represents  the  per¬ 
centage  of  engagements  ending  in  an  AIM-9  kill  by  the  given 
pilot  or  pilots. 

AIM-7  Success:  Does  not  apply  to  the  present  data  since 
no  AIM-7  missiles  were  available  to  the  pilots  during 
similar  aircraft  engagements.  This  calculation  Is  analogous 
to  that  for  measure  Number  23. 


25.  Gun  Success:  The  probability  that  the  pilot  achieved  a  gun 
kill  during  the  given  engagement.  The  calculation  is 
analogous  to  that  for  measure  Number  23. 

26.  Ground  Kill  Success:  The  probability  that  the  pilot 
achieved  a  kill  due  to  his  opponent's  flight  into  terrain 
during  the  given  engagement.  The  calculation  is  analogous 
to  that  for  measure  Number  23. 

27.  Over-g  Success:  The  probability  that  the  pilot  achieved 
a  kill  due  to  his  opponent  overstressing  the  aircraft, 
i.e.,  exceeding  9.5  g,  during  the  given  engagement. 

The  calculation  is  analogous  to  that  for  measure  Number  23. 

28.  Fuel  Kill  Success:  The  probability  that  the  pilot  achieved 
a  kill  due  to  his  opponent  passing  the  "bingo"  fuel  limit 
without  turning  to  his  "home"  heading.  This  calculation  is 
analogous  to  that  for  measure  Number  23. 

Pretest  Exercises 

All  28  of  the  candidate  measures  were  calculated  for  the  pretest 
exercises.  On  the  pretest  exercises,  the  most  striking  feature  of 
these  data  is  a  near  absence  of  consistent  and  explainable  differences 
between  the  three  skill  level  groups  of  pilots.  The  only  noteworthy 
difference  between  the  three  groups  was  in  their  use  of  the  throttle 
control.  Group  1  pilots  tended  to  use  their  afterburner  to  a  greater 
extent  than  pilots  in  the  other  groups.  Groups  1,  2,  and  3  used  the 
afterburner  42%,  38%,  and  28%  of  the  engagement,  respectively.  Conversely, 
Group  3  pilots  spent  more  time  with  their  throttles  at  idle  (15%)  than 
did  Group  1  (11%)  or  Group  2  (8%).  Use  of  the  throttle  control  is 
graphically  depicted  in  Figure  2. 

Consistent  with  the  findings  concerning  throttle  usage,  the  measure 
of  fuel  flow  reflected  the  same  difference  between  the  three  groups. 

As  would  be  expected  from  the  data  on  throttle  use,  Group  3  had  the  low¬ 
est  average  fuel  flow  (22607  lb./hr.).  Group  1  had  the  highest  average 
fuel  flow  (26142  lb./hr.).  The  average  fuel  flow  for  Group  2  pilots 
was  25969  lb./hr.  These  findings  concerning  throttle  use  by  the  three 
groups  were  consistent  with  the  data  recorded  during  the  Ivl  free  engage¬ 
ments  (to  be  discussed  later). 


This  near  absence  of  clear  differences  between  the  pilots  during 
the  pretest  exercises  may  have  strong  implications  for  the  prediction  of 
pilot  performance  during  ACM.  The  pilots  taking  part  in  our  testing  were 
deliberately  chosen  to  represent  widely  different  experience  and  skill 
levels.  The  pretest  consisted  of  standard  exercises  used  to  test  pilot 
ACM  pjrformance  in  training  and  during  operational  readiness  inspection 
exercises.  Yet,  although  the  pretests  were  carefully  controlled  and  the 
results  precisely  examined,  there  was  little  in  all  the  data  collected 
to  indicate  the  group  differences  which  existed. 

The  investigators  believe  that  this  absence  of  group  differences  was 
largely  due  to  the  well  defined  nature  of  the  task.  In  the  first  two 
pretest  exercises,  the  pilots  were  told  which  profile  their  target  would 
be  flying  and  what  actions  were  expected  of  them.  All  pilots,  therefore, 
flew  a  standardized  attack  profile  which  eliminated  the  need  for  much  of 
the  rapid  decision-making  which  characterizes  free  engagements.  This  was 
also  true  to  a  lesser  extent  during  the  three  attacks  which  comprised 
Maneuver  III  on  the  pretest.  The  pilots  realized  that  their  target  was 
merely  flying  a  predetermined  pattern  rather  than  responding  to  their 
actions.  There  was  no  need  for  counteroffensive  maneuvering  which  greatly 
decreased  the  cognitive  requirements  placed  on  the  pilots  and  limited 
their  task  almost  to  a  simple  BFM  problem.  Thus,  the  highly  skilled  and 
experienced  pilots  were  unable  to  take  full  advantage  of  their  greater 
experience  and  skill  during  the  pretest.  Most  of  the  group  differences 
were  obscured  by  the  differences  between  individuals  within  the  groups. 

Univariate  Analysis  of  Free  Engagements 

By  contrast,  the  1 vl  free  engagements  allowed  the  obvious  group 
differences  to  be  demonstrated.  The  whole  engagement  performance 
measures  provided  clear  and  predictable  indications  of  group  differences 
for  nearly  every  parameter.  Table  4  shows  the  mean  values  and  standard 
deviations  for  all  the  whole  engagement  measures,  most  of  which  are 
discussed  in  the  following: 

Throttle  Use.  Clear  differences  were  found  between  the  groups  in 
their  use  of  the  throttle.  These  differences  were  in  the  same  direction 
as  those  found  in  the  pretest  data  but  were  much  more  pronounced  (see 
Figure  3).  As  skill  level  increased  there  was  a  definite  tendency  to 
use  the  afterburner  less  and  the  idle  position  more. 

Fuel  Flow.  Because  of  the  tendency  for  the  lower  skilled  pilots 
to  use  higher  power  settings  during  the  engagements,  it  was  predictable 
that  their  average  fuel  flow  should  be  higher  than  that  or  the  more 
skilled  pilots.  This,  indeed,  was  the  case.  For  each  of  the  three  init¬ 
ial  setups  flown  by  the  pilots,  Group  1  had  the  highest  average  fuel 
flow  and  Group  3  had  the  lowest. 
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Figure  3.  Throttle  Use  During  'VI 


Airspeed.  Similarly,  the  data  on  throttle  use  would  suggest  that 
the  lower  skilled  pilots  fought  at  a  higher  airspeed,  on  the  average, 
than  did  the  more  highly  skilled  pilots.  Table  4  shows  this  to  be  the 
case.  This  difference,  in  itself,  provided  an  advantage  to  the  more 
highly  skilled  pilots  with  the  lower  airspeeds  allowing  tighter  turns 
during  the  engagement.  This  is  in  agreement  with  the  predictions  of 
the  subject  matter  experts  who  suggested  that  the  more  aggressive  pilots 
operate  at  lower  airspeeds  in  order  to  maximize  the  probability  of  a 
kill,  even  though  this  increases  the  chance  that  they,  themselves,  will 
be  killed. 

Energy  Management  Index,  Another  measure  which  indicated  a  differ¬ 
ence  between  the  three  skill  level  groups  during  the  free  engagements 
was  the  Energy  Management  Index  (EMI),  This  finding  confirmed  previous 
speculation  from  many  sources  that  the  skilled  ACM  pilot  manages  his 
energy  more  efficiently  than  does  the  less  skilled  pilot.  Note  that  in 
every  one  of  the  initial  setups  Group  3  pilots  had  the  highest  EMI  score 
followed  by  Group  2  pilots  and  then  Group  1  pilots. 

Speedbrake.  The  more  skilled  pilots  tended  to  use  the  speedbrake  to 
a  greater  extent  than  did  the  less  skilled  pilots.  It  is  important,  how¬ 
ever,  to  note  the  highly  skewed  nature  of  the  distributions.  It  is 
apparent  from  a  comparison  of  the  means  and  standard  deviations  that  most 
of  the  pilots  in  all  three  groups  used  the  speedbrake  very  little.  The 
relatively  small  means  and  large  standard  deviations  indicate  that  most 
of  the  speedbrake  use  was  done  by  a  small  percentage  of  the  pilots. 

Altitude  Rate.  Altitude  rate  is  the  mean  absolute  vertical  speed 
that  a  pilot  maintains  during  an  engagement.  This  is  one  means  of  assess¬ 
ing  the  degree  to  which  a  pilot  uses  the  vertical  dimension  of  the  air¬ 
space  during  an  engagement.  A  pilot  who  makes  little  use  of  the  vertical 
will, of  course,  ha''e  a  low  value  for  the  average  altitude  rate  while  a 
pilot  who  makes  greater  use  of  the  vertical  will  tend  to  have  a  higher 
value.  Because  it  has  been  assumed  by  many  authorities  that  the  more 
skilled  and  successful  ACM  pilot  makes  greater  use  of  the  vertical  dimen¬ 
sion,  the  investigators  expected  the  Group  3  pilots  have  the  highest 
value  for  altitude  rate.  We  found,  however,  just  the  opposite  trend. 

The  lower  skilled  Group  1  snowed  the  highest  average  altitude 

rate  (276  ft. /min.)  while  Group  3  pilots  had  the  lowest  value  (244  ft./ 

min.)  with  Group  2  in  between  (256  ft./min.). 

Relative  Altitude  Use.  A  related  measure  is  the  relative  altitude  use 
which  compared  the  amount  of  vertical  airspace  used  by  the  pilot  with  the 
amount  used  by  his  opponent.  These  data  reflected  the  same  trend  as  the 
data  on  altitude  rate.  The  ratios  calculated  for  Groups  1,  2,  and  3  were 
1.07,  1.01,  and  .99,  respectively.  While  these  differences  were  not  large, 
they  provided  further  evidence  that,  at  least  in  this  study,  the  lower 
skilled  pilots  made  at  least  as  much  use  of  the  vertical  dimension  as  their 
more  highly  skilled  counterparts. 
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There  is  one  possible  explanation  for  this  unexpected  finding  which 
deserves  future  exploration.  Pilots  in  Group  1  were  killed  by  terrain 
contact  approximately  four  times  as  often  as  pilots  in  Group  3.  It  is 
possible  that  much  of  the  difference  between  groups  found  here  is  the 
result  of  high  speed,  uncontrolled  descents  by  the  less  skilled  pilots. 

Future  examinations  of  the  data  should  investigate  this  possibility. 

Average  Roll  Rate.  The  investigators  expected  that  the  skilled 
pilots  would  make  more  use  of  the  roll  axis  of  the  aircraft  than  would 
their  less  skilled  counterparts.  This  would  have  been  reflected  in  the 
data  on  absolute  average  roll  rate  with  the  skilled  pilots  showing  a 
higher  rate.  While  this  prediction  was  not  confirmed  by  these  univariate 
analyses,  they  were  confirmed  by  later  multivariate  analyses  Of  the  measures. 

Angle  of  Attack.  Pilots  exceeding  28  units  AOA  are  approaching  loss 
of  control.  The  investigators  expected  that  during  ACM,  when  pilots 
are  maneuvering  in  high  angle  of  attack  regimes,  the  pilots  more  highly 
skilled  in  aircraft  maneuvering  would  approach,  but  not  surpass,  this 
value.  There  was,  however,  little  or  no  difference  between  the  skill 
level  groups  in  the  percentage  of  time  spent  with  an  AOA  of  more  chan 
28  units.  Pilots  in  all  three  skill  level  groups  spent  approximately 
one  percent  of  their  engagement  time  at  more  than  28  units  ACA. 

Offensive  Time.  It  is  logical  to  expect  pilots  of  higher  skill  level 
to  spend  a  larger  percentage  of  the  engagement  in  an  offensive  posture 
than  less  skilled  pilots.  This  expected  trend  was  clearly  shown  in  our 
data.  Pilots  in  Group  3  spent  more  than  twice  as  much  time  in  the  offen¬ 
sive  as  did  Group  1  pilots.  Group  2  pilots  fell  approximately  halfway 
between  the  other  groups. 

Offensive  Time  with  Advantage.  The  above  measure  of  offensive  time 
is  simply  a  measure  of  the  percentage  of  the  engagement  the  pilot  is 
pointed  toward  his  opponent.  It  is  possible  for  both  pilots  to  be  offen¬ 
sive  simultaneously.  This  occurs,  for  example,  when  both  are  approaching 
for  a  head  on  gun  pass.  By  contrast,  to  be  offensive  with  advantage, 
the  pilot  had  to  be  pointed  in  the  direction  of  his  opponent's  tail. 

Again,  the  investigators  expected  the  Group  3  pilots  to  accrue  more  time 
in  an  offensive  with  advantage  position  than  the  less  skilled  pilots. 

Indeed,  Group  3  pilots  spent  about  25%  of  their  engagements  offensive 
with  advantage  compared  to  8%  and  16%  for  Groups  1  and  2,  respectively. 

Time  in  Range.  This  is  the  amount  of  time  spent  within  gun  range 
during  an  engagement.  As  expected  there  is  a  strong  and  consistent 
relationship  between  the  skill  level  of  the  pilot  and  the  amount  of  time 
spent  within  gun  range. 
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Out  of  View  Time..  Directly  related  to  the  measures  of  offensive 
time  and  offensive  with  advantage  is  the  out  of  view  time.  Ideally, 
it  could  be  expected  that  highly  skilled  pilots  would  spend  a  large 
percentage  of  the  engagement  out  of  their  opponent's  field  of  view.  The 
less  skilled  pilot  could  be  expected  to  spend  a  smaller  portion  of  time 
out  of  his  opponent's  view.  This  was,  indeed,  found  to  be  the  case. 

Group  3  pilots  spent  about  a  third  more  time  out  of  view  than  did  the 
Group  1  pilots.  The  out  of  view  time  for  Group  2  pilots  was  midway 
between  those  found  for  the  other  two  groups. 

Lead  Pursuit  Time.  A  pilot  employing  lead  pursuit  to  intercept 
his  opponent  can  enter  the  fight  more  quickly  than  a  pilot  employing  pure 
pursuit  or  lag  pursuit.  Lead  pursuit  time,  therefore,  may  correlate  with 
aggressiveness.  The  investigators  expected  the  less  skilled  pilots  to 
spend  a  smaller  portion  the  engagement  in  lead  pursuit  than  their  more 
skilled  counterparts.  Group  1  pilots  actually  spent  about  half  as  much 
time  in  lead  pursuit  as  Group  2  pilots  and  a  third  as  much  as  Group  3 
pilots. 

Defensive  Time.  Since  the  more  skilled  pilots  tended  to  spend  a 
larger  percentage  of  their  engagement  time  on  the  offensive,  it  was 
reasonable  to  assume  that  they  might  spend  a  smaller  percentage  of  time 
in  a  defensive  position.  The  data  confirmed  this  assumption. 

Mission  Success.  The  success  of  an  ACM  mission  is  generally  deter¬ 
mined  by  the  kills  occurring  during  that  mission.  For  that  reason  some 
emphasis  was  placed  on  examining  the  numbers  and  kinds  of  kills  scored 
by  the  pilots.  Summaries  of  kills  using  AIM-9  missiles,  guns,  ground 
kills,  over-g  kills,  and  out  of  fuel  kills  showed  a  considerable  effect 
of  skill  level  on  the  numbers  and  types  of  kills  scored.  These  data  are 
summarized  in  Figure  4. 

It  was  apparent  that  success  with  weapons  was  an  excellent  indicator 
of  skill  level.  The  success  ratio,  using  missiles,  for  Group  3  pilots 
was  more  than  double  that  of  Group  2  pilots  and  about  four  times  that  of 
Group  1  pilots.  The  differences  between  the  groups  was  even  more  marked 
for  the  data  on  gun  success.  Group  3  pilots  had  about  three  times  the 
number  of  gun  kills  scored  by  Group  2  pilots  while  Group  1  pilots  did  not 
score  a  gun  kill.  The  combined  success  with  both  missiles  and  guns  again 
showed  a  strong  difference  between  the  three  groups  of  pilots. 

Kills  scored  by  other  than  weapons  delivery  showed  less  promise  as  an 
indicator  of  ACM  performance.  The  data  indicated  only  small  differences 
between  the  groups  in  kills  due  to  ground  impact,  over-g  and  fuel  depletion. 
These  outcomes  appeared  to  be  influenced  by  chance  to  a  much  greater 
degree  than  did  the  weapons  kills  and  are  probably  of  little  value  in 
measurement  of  ACM  performance. 
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Other  Measures.  Data  analyses  Indicated  that  the  measures  of  Abso¬ 
lute  Average  Heading,  RMS  Heading,  Roll  Rate  times  Altitude  Rate,  and 
Plane  of  Action,  as  calculated  using  the  initial  algorithms,  did  not 
discriminate  between  pilots  of  different  skill  levels. 

Multivariate  Analyses  of  Whole  Engagement  Measures 

The  univariate  analyses  of  whole  engagement  measures  indicated  that 
these  measures  could  provide  the  basis  for  an  ACMPM  system  which  would 
discriminate  between  pilots  on  the  basis  of  ACM  skill  level.  These 
data,  however,  had  too  much  underlying  variance  within  Groups  to  be 
employed  in  a  univariate  manner  in  an  ACMPM  system.  Because  there  was 
a  high  degree  of  correlation  between  many  of  the  measures,  the  investiga¬ 
tors  employed  multivariate  analyses  in  an  attempt  to  reduce  this  unexplained 
variance  and  to  improve  the  efficacy  of  the  ACMPM  structure. 

Because  this  effort  represented  a  preliminary  study  to  determine  the 
feasibility  of  developing  an  ACMPM  system,  the  investigators  determined 
that  a  structure  that  could  discriminate  between  the  two  most  diverse 
skill  levels  of  pilots  would  serve  this  purpose.  The  multivariate  measures 
were,  therefore,  applied  to  data  representing  engagements  flown  by  (a) 

Group  1  pilots  versus  pilots  from  the  higher  skill  groups  and  (b)  Group  3 
pilots  versus  pilots  from  the  lower  skill  level  groups. 

Candidate  Measures.  The  multivariate  analyses  began  with  31  whole 
engagement  variables.  These  included  the  measures  examined  during  the 
univariate  analyses  described  above  with  several  additions  and  modifica¬ 
tions  which  the  investigators  believed  might  account  for  portions  of  the 
total  variance.  These  additions  included  measures  accounting  for  any 
learning  effect  and  for  any  difference  between  the  three  setup  conditions 
which  started  the  engagements.  To  account  for  a  possible  learning  effect 
a  variable  representing  the  position  of  the  engagement  in  the  sequence  of 
15  engagements  flown  by  each  pilot  was  included.  To  account  for  differ¬ 
ences  due  to  initial  setup,  dummy  variables  were  included  to  flag  data 
from  engagements  initialized  at  each  starting  position. 

In  addition,  minor  changes  were  made  in  the  algorithms  for  calculating 
two  candidate  measures  to  improve  the  discrimination  between  pilot  skill 
level.  First,  the  algorithm  for  determining  opponent  out  of  view  was 
calculated  as  shown  in  Figure  C8,  rather  than  using  the  simulation  param¬ 
eter  for  out  of  view.  Second,  counting  transitions  into  each  throttle 
zone  was  found  to  be  more  discriminating  than  the  time  in  each  zone.  The 
number  of  transitions  into  each  throttle  zone  was  divided  by  the  total 
number  of  data  samples  to  form  the  revised  throttle  activity  measures. 

All  31  candidate  measures  for  the  initial  multivariate  analysis  are 
presented  in  Table  5,  More  complete  definitions  of  the  algorithms  are 
contained  in  Appendix  C. 
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Data  Editing.  The  first  step  in  the  preparation  of  data  for  analysis 
was  to  sort  the  data  into  two  groups.  Group  A  was  composed  of  data  from 
relatively  inexperienced  pilots  (Group  1)  when  they  were  fighting  more 
experienced  pilots  (Group  2  and  Group  3).  Group  B  was  composed  of  data 
from  the  most  experienced  pilots  (Group  3)  when  they  were  fighting  less 
experienced  pilots  (Group  2  and  Group  1). 

The  next  step  was  to  remove  useless  leading  and  trailing  data  tape 
records  which  were  caused  by  manual  control  of  recording  during  data 
collection.  Computer  logic  was  written  to  detect  and  remove  data  prior 
to  engagement  start  and  after  its  termination. 

The  third  step  was  to  remove  uncharacteristic  performance  from  the 
data  of  the  undesireable  effects  of  a  few  highly  variant  performances  on 
a  regression  analysis.  These  atypical  observations,  called  outliers,  were 
detected  by  looking  for  any  single  measure  which  was  more  than  3.5a  away 
from  that  groups'  mean  for  that  particular  measure.  Table  6  is  a  tally 
of  the  measures  for  each  group  that  were  found  to  be  outliers.  There 
were  130  observations  for  each  group,  but  after  outlier  removal  there 
were  112  in  Group  A  and  115  in  Group  B.  The  group  means  and  standard 
deviations  for  the  remaining  measures  may  be  found  in  Appendix  E. 

A  fourth  step  was  necessary  to  look  closely  at  the  group  correla¬ 
tion  matrices  (also  in  Appendix  E)  because  highly  correlated  measures 
will  have  a  negative  effect  on  the  discriminant  analysis.  A  program  was 
written  which  would  identify  couplets  of  highly  correlated  measures  and 
determine  which  of  the  measures  in  the  pair  was  most  correlated  to  all 
the  other  measures  in  their  group.  Six  measures  were  eliminated  from 
analysis  as  they  correlated  greater  than  .83  with  some  other  measure. 

The  results  of  this  step  are  shown  in  Table  7.  The  criterion  for  elimina¬ 
tion  was  arbitrary  and  it  left  25  measures  for  further  analysis. 

Data  Analysis.  The  results  reported  in  this  section  are  but  the 
highlights  of  several  multivariate  analyses  of  the  ACM  data.  The  final 
model  was  developed  through  many  successive  approximations  guided  by  the 
analysts'  decisions  regarding  tradeoffs  between  group  discrimination  and 
accounted  for  variance  in  the  model.  Five  to  ten  percent  of  the  variance 
was  sacrificed  for  the  sake  of  discriminability  and  other  pragmatic 
criteria.  Linear  relationships  could  be  found  which  accounted  for  67.5% 
of  the  variance  but  relied  very  heavily  on  energy  control  measures.  Practi¬ 
cally,  the  performance  measurement  model  had  to  include  several  other 
aspects  of  the  ACM  task.  With  this  in  mind,  the  analysis  algorithm  could 
not  be  given  free  rein,  but  had  to  be  carefully  controlled  at  all  stages. 


Measure 

Number 

Measure 

Name2 

Group  1 
Frequency1 

Group  2 
Frequency 

3 

ASRA 

1 

4 

SBPA 

2 

3 

5 

FFA 

1 

m 

7 

EMRM 

5 

- 

9 

SADP 

2 

- 

10 

TH1A 

2 

- 

11 

TH2A 

1 

1 

12 

TH3A 

1 

2 

14 

HDGRAA 

1 

- 

17 

IRP 

1 

2 

18 

RRAA 

1 

- 

19 

RRARA 

- 

1 

21 

DDP 

1 

3 

22 

A28P 

- 

2 

Total 

Observations 

Removed 

18 

15 

Frequency  is  the  number  of  times  in  a  group  that  a  particul 
measure  was  found  to  be  more  than  3.5a  outside  the  group 
membership. 


See  Te’'1  *•  for  definitions 


TABLE  7.  HIGHLY  CORRELATED  MEASURES  REMOVED 


1st  Variable 
in  Pair 

2nd  Variable 
in  Pair 

Measure 

Number 

Measure 

Name' 

Measure 

Number 

Measure 

Name 

Correlation 

Variable 

Removed 

8 

OFFP 

16 

LEADP 

.86 

8 

9 

SADP 

29 

L/0 

.97 

29 

9 

SADP 

16 

LEADP 

.99 

9 

11 

TH2A 

13 

TH4A 

.82 

11 

14 

HDGRM 

15 

HDGRAA 

.95 

15 

18 

RRAA 

19 

RRARA 

.84 

19 

^ee  Table  5  for  definitions. 


Several  stepwise  regression  and  discriminant  analyses  were  performed 
in  search  of  a  useable  measurement  model.  Although  many  useful  insights 
about  the  available  data  were  gained,  a  practical  performance  model  was 
not  immediately  revealed  by  these  stepwise  procedures.  Being  stepwise, 
the  results  of  these  programs  were  dependent  on  the  decisions,  related 
to  the  F  level  or  communal i ties-  made  early  in  the  selection  or  deletion 
of  measures.  Different  measure  sets,  of  course,  were  selected  by  the 
programs.  This  provided  a  hint  at  the  common  set  that  would  give  practi¬ 
cal  utility. 

A  regression  analysis  of  the  entire  measure  set  was  performed  and 
can  be  seen  in  Table  8.  In  Table  8  the  assigned  number  relates  to  the 
measures  remaining  after  the  data  editing  stage.  Based  on  the  appearance 
of  certain  measures  in  previous  analyses  and  the  significance  test  in 
the  regression  analysis,  16  final  candidate  measures  were  selected.  It 
was  felt  that  these  measures  would  be  the  best  starting  point  for  the 
final  analyses,  usina  a  step  down  discriminant  analysis. 

Table  9  shows  the  correlation  matrix  of  the  16  remaining  candidate 
measures.  Table  10  shows  their  correlations  to  group  membership. 
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P  (8)  was  not  included  in  complete  regression  set  because  of  high 
with  LEADP  (16).  Since  LEADP  (16)  was  not  included  in  DISCRIM  set 
>e  included  without  conflict,  and  the  means  and  S.D.  are  included 
*  reader's  information. 


Table  11  shows  the  discriminant  analysis  results.  The  assigned 
number  relates  purely  to  those  16  candidate  measures  in  the  analysis 
as  an  identifier.  The  first  column  of  coefficients  and  conmunalities  is 
for  the  complete  group  with  all  measures  present.  The  second  column  is 
with  AIM9P  and  INIT3  removed.  These  two  measures  were  eliminated  on  the 
basis  that  their  communal i ties  were  less  than  .0025.  The  third  column 
shows  the  final  discriminant  analysis  of  13  measures.  Measure  "g"  was 
eliminated  on  the  basis  of  low  communal ity. 

A  ridge  adjusted  discriminant  analysis  (Vreuls,  et  al^,  1976)  was 
performed  on  the  final  13  measures.  The  ridge  procedure  removes  some 
of  the  "oveir-f»t"  of  a  linear  discriminant  model  and  makes  subsequent  appli¬ 
cations  of  the  model  more  reliable.  The  results  are  shown  in  Table  12. 

As  bias  (K  r  .1,  K  =  .2,  etc.)  was  added  to  the  diagonal  of  the  inter¬ 
correlation  matrix  prior  to  successive  discriminant  analyses,  the  linear 
combination  of  the  coefficients  for  each  measure  tends  to  stabilize. 

This  is  illustrated  also  in  Figure  5  for  five  selected  measures. 

Test  of  Resulting  Model.  The  discrimant  measurement  models  were 
tested  using  Monte  Carlo  type  data  simulations  to  explore  their  relative 
potential  reliability.  With  the  aid  of  this  data  simulation  technique 
and  the  changes  in  the  coefficient  trace  of  the  ridge  adjusted  discrimi¬ 
nant  analysis,  the  most  potentially  reliable  model  could  be  selected  by 
choosing  the  model  which  revealed  the  minimum  misclassification  error. 

The  resultant  model  had  a  K  value  of  0,3  and  a  total  average  misclassifi¬ 
cation  error  of  16%.  The  complete  results  of  these  tests  are  presented 
in  Appendix  F. 

Summary  of  Multivariate  Analysis  Results.  The  multivariate  analyses 
of  time  history  data  collected  during  the  Ivl  ACM  free  engagements 
demonstrated  the  feasibility  of  an  ACMPM  structure  based  on  such  data. 

The  results  of  a  13  measure  discriminant  analysis  provided  an  effective 
model.  While  the  optimum  value  for  K  was  not  determined,  a  value  of  0.3 
was  found  to  be  a  satisfactory  approximation. 

Given  that  a  pilot  flies  the  F-4  in  the  SAAC  simulator  in  the  same 
configuration  as  this  study,  and  giyen  the  same  initializations  before 
release,  and  under  the  same  rules  and  restrictions,  the  criterion  of 
performance  is  based  on  the  algo  chms  presented  in  Table  13. 

If  the  score  is  greater  than  -1.8999  there  is  a  probability  greater 
than  0.921  that  the  pilot  belongs  to  the  criterion  group  of  experienced 
pilots  (analysis  Group  3).  The  criterion  can  be  adjusted  by  tuning  the 
cut  off  score  about  the  group  mean,  -1.1438.  To  be  absolutely  sure  of 
the  decisions  made  by  this  scoring  model,  some  3.5a  negative  extrema 
tests  for  particular  parameters  should  be  made  before  scoring.  This 
would  insure  that  the  pilot  performance  being  scored  reasonably  belongs 
to  a  sample  group  from  which  the  scoring  model  was  developed. 
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TABLE  13.  PERFORMANCE  ALGORITHM  FOR  lvl  ON  Sh*C 


Weighting 

Coefficient 

Measure 

-  .00529 

Absolute  Average  Altitude  Rate 

-  .41231 

Percent  Opponent  Out  of  View 

+  .05922 

Average  Speed  Brake  Position 

-  .00006 

Average  Fuel  Flow 

+  .01212 

RMS  Energy  Management  Index 

+1.42378 

Percent  Offensive 

+2.27077 

Average  Throttle  Position  1  Transitions 

-1.20926 

Average  Throttle  Position  3  Transitions 

+2.68293 

Average  Throttle  Position  4  Transitions 

+3.05355 

Percent  in  Range 

+  .01131 

Absolute  Average  Roll  Rate 

+  .55229 

1.0  if  Initialized  in  Position  2 

+  .62546 

1.0  if  Successful  Gun  Kill 

C. 

Tactical  Level  Performance  Measurement 

During  training  in  BFM,  pilots  are  taught  maneuvering  rules  for 
various  combinations  of  range,  range  rate,  line-of-sight  angle,  and 
aspect  angle  between  their  aircraft  and  the  target  aircraft.  The 
tactical  level  analyses  are  based  on  the  assumption  that  range,  LOS,  and 
aspect  angle  are  important  variables  in  determining  the  proper  course 
of  action  for  each  pilot  and  for  determining  the  outcome  of  the 
engagement. 


A  three  dimensional  structure  (TACSPACE)  was  imposed  on  the  time 
history  data  from  the  lvl  free  engagements.  TACSPACE  is  best  described 
as  sectors  of  a  volume  with  LOS  angle,  aspect  angle  and  range  as  the 
three  axes  (see  Figure  6).  Note  that  the  scales  on  the  three  axes  are 
not  linear  since  the  sectors  were  dimensioned  to  relate  to  expected 
population  densities. 
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The  initial  tactical  analysis  tested  the  hypothesis  that  a  relation¬ 
ship  existed  between  the  various  parts  of  TACSPACE  used  by  pilots  and 
the  outcome  of  the  engagement  (winner  or  loser).  This  hypothesis 
received  some  support  from  relative  aircraft  position  data  (e.g.,  offen¬ 
sive  time,  out  of  view  time)  in  the  univariate  analyses.  The  TACSPACE 
analysis,  however,  examined  the  relative  position  data  with  somewhat 
greater  precision. 

TACSPACE  data  were  examined  for  the  winners  and  losers  of  all 
engagements  in  which  a  weapons  kill  was  accomplished.  The  percentage  of 
engagement  time  the  pilots  spent  in  each  of  the  125  TACSPACE  "boxes"  was 
calculated.  The  "population"  in  each  box  was  scaled  by  a  population 
scaling  factor  (PSF)  to  compensate  for  unequal  box  volumes  of  TACSPACE. 

The  boxes  were  ranked  according  to  the  percentage  of  mission  time  spent  in 
each  box  by  the  winners  and  by  the  losers.  The  distributions  of  the  result 
ant  corrected  box  mission  times  are  shown  in  Tables  G1  and  G2. 

Note  that  the  distribution  is  symmetrical  as  the  "most  winning" 
sequence  of  boxes,  starting  with  Box  122,  degrades  toward  the  "most 
losing"  box  (452).  At  this  stage,  it  was  decided  to  further  examine 
the  top  10%  of  winner  and  loser  boxes  since  these  probably  contained  the 
most  descriptive  significance.  Table  G3  shows  that,  with  two  exceptions, 
the  losing  box  was  the  "reciprocal"  of  the  winning  box.  This  suggests 
that  the  percentage  of  time  each  pilot  spent  in  each  of  these  boxes  could 
be  used  as  a  measure  of  performance. 

These  preliminary  TACSPACE  data  provide  only  a  suggestion  of  the 
potential  utility  of  the  TACSPACE  approach  to  ACMPM.  Future  analyses 
might  examine  the  way  in  which  pilots  fly  their  aircraft  as  a  function 
of  their  position  in  TACSPACE.  Preliminary  glances  at  these  data  suggest 
interesting  trends.  For  example,  when  highly  skilled  pilots  were  in  a 
position  of  small  aspect  and  LOS  angles,  they  were  very  active  in  throttle 
and  speedbrake  control.  The  lower  skilled  pilots  became  so  busy  tracking 
their  opponents  that  their  throttle  and  speedbrake  activity  approached 
zero, 

Additionally,  when  high  skilled  pilots  were  in  the  offensive  with 
advantage  position,  their  mean  g-loading  was  significantly  lower  than 
that  of  the  lower  skilled  pilots  when  in  similar  positions.  Conversely, 
when  the  highly  skilled  pilots  were  defensive  with  disadvantage,  they 
used  considerably  higher  g-loading  during  their  counteroffensive  maneuver¬ 
ing  than  did  their  lower  skilled  counterparts. 

These  initial  examinations  of  the  TACSPACE  data  base  indicate  that 
this  approach  can  provide  information  of  considerable  diagnostic  value 
when  the  TACSPACE  concept  is  fully  developed  and  refined. 


V.  CONCLUSIONS 


This  study  represents  the  development  and  preliminary  analysis  of 
a  fertile  body  of  data  collected  during  lvl  ACM  free  engagements.  The 
data  collected  and  the  measurement  structures  developed  contain  the 
seeds  of  an  effective  ACMPM  system  which,  when  further  refined,  could 
be  implemented  on  the  SAAC  and,  with  some  revision,  on  the  ACMR/ACMI. 

Many  measures,  taken  at  the  whole  engagement  level,  were  found  to 
correlate  with  pilot  ACM  skill  level.  For  example,  the  highly  skilled 
pilots  achieved  more  kills,  used  less  fuel,  flew  slower  speeds,  and 
were  more  active  in  throttle  and  speedbrake  control  than  the  pilots  of 
lesser  skill.  All  of  these  individual  measures,  however,  contained  large 
amounts  of  underlying  within-groups  variance  which  precluded  the  use  of 
any  single  measure  as  a  reliable  metric  of  ACM  skill  level. 

In  order  to  minimize  this  unexplained  variance,  multivariate  anal¬ 
ysis  yielded  a  measurement  model  containing  13  variables  including  meas¬ 
ures  of  energy  management,  control  activity,  relative  aircraft  positions, 
aircraft  maneuvering,  and  success  with  guns.  This  model,  with  optimal 
weights  placed  on  each  of  the  measures,  accounted  for  51%  of  the  perform¬ 
ance  variance.  It  was  able  to  discriminate  between  pilots  of  high  skill 
and  pilots  of  lower  skill  with  an  accuracy  of  92.1  percent.  Future 
iterations  of  the  analysis  process  and  the  addition  of  diagnostic  meas¬ 
ures  could  be  expected  to  further  refine  the  measurement  structure  and 
improve  this  accuracy. 

The  tactics  analysis  represents  a  potentially  useful  way  of  examin¬ 
ing  performance  within  an  engagement  for  the  purpose  of  suggesting  diag¬ 
nostic  measures.  These  data,  however,  are  very  preliminary  and  the 
TACSPACE  concept  requires  considerable  development  before  much  useful 
information  can  be  derived  from  it.  The  investigators  believe  that  this 
further  development  will  demonstrate  that  the  TACSPACE  concept  has  great 
potential  as  an  ACMPM  diagnostic  structure. 

It  is  important  to  recognize  that  the  measurement  structures  and 
algorithms  developed  here  are  very  application  specific.  They  apply 
only  to  lvl  free  engagements  flown  on  the  SAAC  in  an  F-4  versus  F-4  mode 
using  the  three  initial  setups  tested  here.  It  is  expected  that,  with 
some  revision,  the  model  could  be  applied  in  other  ACM  environments. 

Any  attempt  to  do  so  without  prior  empirical  evaluation  of  the  model  in 
the  new  application  would  be  extremely  ill-advised,  especially  since 
it  is  relatively  straightforward  to  perform  empirical'  tests. 


The  major  investment  required  to  perform  future  empirical  tests 
will  be  in  the  data  collection  activities.  The  basic  approach  and 
methods  to  analyze  the  data  for  measurement  model  development  have  been 
developed.  Thus,  future  analyses  of  new  data  to  improve  the  model  for 
changes  in  task  conditions  can  be  performed  with  a  minimum  level  of 
effort  and  in  a  very  short  time. 

It  must  be  emphasized  that  none  of  the  ACMPM  structures  developed 
during  this  effort  represents  a  finished  product  ready  for  implementation 
and  operational  use.  Further  analyses,  total  system  design  and  develop¬ 
ment  activities,  developmental  testing  and  validation  testing  should  be 
undertaken  along  the  lines  suggested  in  the  next  section. 


*  m  „ 

a&W  ufc.  id s£ i£tf rfr ^ ^ -  +*■ 


rTTrr  j-.  n>  'Vi-y  3 


VI.  RECOMMENDATIONS 


This  study  achieved  its  goals  by  providing  an  empirical  data  base, 
a  methodology,  and  an  initial  algorithm  for  measuring  Ivl  ACM  performance. 
Several  further  steps  are  required,  however,  to  produce  an  effective 
ACMPM  system  which  will  be  useable  by  instructor  pilots  on  the  SAAC: 

1.  The  measure  set  should  be  refined  through  iterative  computer 
analyses  of  the  existing  data  base.  First,  an  analysis  of  dissimilar 
aircraft  engagements  should  be  conducted  to  develop  a  measurement  model 
which  is  appropriate  for  F-4  versus  MIG  engagements.  Second,  the  diag¬ 
nostic  measurement  capability  simplification  of  TACSPACE  should  be 
explored.  Finally,  the  results  of  these  analyses  and  other  ACMPM  efforts 
should  be  incorporated  in  the  measurement  structure  where  appropriate, 
and  final  algorithms  developed. 

2.  The  algorithms  should  be  cross-validated  with  a  new  sample  of 
pilots  on  the  SAAC.  The  Monte. Carlo  type  data  simulation  which  was 
performed  in  this  study  represents  a  good  initial  test  of  the  measurement 
model.  A  cross-validation,  howevet ,  would  represent  a  more  complete  and 
final  test. 

3.  The  design  requirements  for  the  total,  real-time  ACMPM  system 
should  be  specified.  This  includes  a  description  of  the  measurement 
structures  and  algorithms,  the  user  interface  (what  data  are  provided 
to  the  instructors  and  how  they  are  presented),  instructional  materials 
to  aid  the  instructor  in  interpreting  the  data,  and  the  possible  design 
and  use  of  the  data  for  a  training  management  system. 

4.  Given  a  complete  specification,  the  total  ACMPM  system  can  be 
designed,  built  and  implemented  for  developmental  testing  of  the  measures, 
user  interface,  instructional  materials  and  training  management  data 
base.  Developmental  testing  w.:i  oe  required  to  discover  and  correct  any 
unanticipated  problems  with  the  system. 

Other  recommendations  are  offered.  These  are  less  intimately  related 
to  the  development  of  the  ACMPM  system  for  the  SAAC,  but  may  have  an 
impact  on  the  development  of  measurement  for  ACMR/ACMI  and  on  future 
evaluation  and  training  techniques: 

5.  In  parallel  with  Steps  3  and  4  above,  a  computer  reanalysis  of 
the  data  base  should  be  undertaken  with  a  reduced  measure  set  which  is 
limited  to  the  data  that  are  readily  available  on  the  ACMR/ACMI.  The 
purpose  of  this  analysis  will  be  to  offer  the  seeds  of  a  measurement 
model  which  can  be  used  for  airborne  performance  evaluation  and  can  be 
used  for  direct  comparison  of  range  data  and  simulator  data  for  training 
effectiveness  evaluations.  As  a  part  of  this  analysis,  a  thorough 
examination  is  required  of  all  current  measurement  and  related  studies  on 
the  ACMR  and  ACMI.  There  were  several  in  progress  during  the  conduct 

of  this  study.  Their  results  might  be  important  in  future  work. 
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6.  To  be  completely  valid,  especially  for  applications  in  the  air, 
the  final  ACMPM  system  should  include  provisions  for  measuring  the 
performance  of  the  WSO.  In  the  SAAC,  WSO  functions  are  automated.  Much 
of  the  system  performance  variance,  which  can  be  expected  to  be  increased 
with  a  human  operating  this  crew  position,  has  been  held  constant  in  the 
SAAC. 

7.  The  use  of  elaborate  ACMPM  structures  to  measure  attacks  on  a 
non-reactive  target  should  be  examined  closely.  Detailed  performance 
measures  may  diagnose  pilot  performance  against  a  non-reactive  target  and 
might  be  useful  for  that  purpose;  but,  detailed  measures  do  not  predict 
free  engagement  performance.  One  reason  might  be  that  ACM  against  a 
non-reactive  target  does  not  permit  the  pilot  to  demonstrate  the  full 
range  of  perceptual  and  cognitive  skills  required  in  free  engagement.  Of 
the  measures  taken  in  this  study,  against  a  non-reactive  target,  the  only 
one  which  appears  to  predict  free  engagement  is  time  to  kill. 

8.  It  must  be  emphasized  that  no  ACMPM  system  should  be  implemented 
and  put  to  operational  use  until  a  thorough  empirical  evaluation  of  its 
efficacy  is  performed. 
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APPENDIX  A: 


EXPERIMENTAL  CONDITIONS  AND  DATA  COLLECTION  FORMS 


TABLE  AT .  INITIAL  CONDITIONS  FOR  MANEUVER  I 


Variables 

Units 

Cockpit  1 

Cockpit  2 

Indicated  Speed 

( Knots  1 

400 

400 

Altitude 

(feet) 

10,000 

10,000 

Heading 

(degrees) 

0 

0 

Line  of  Sight  Angle 

(degrees) 

0 

0 

Slant  Range 

Relative  Horizontal 
Offset 

Cannon  Rouncis 

(feet) 

(#) 

640 

4,000 

0 

640 

AIM-9  (Heat)  Missiles 

(#) 

0 

0 

AIM-7  (Radar)  Missies 

(#) 

0 

0 

Fuel  Quantity 

(Lbs.) 

8,000 

8,000 

Bingo  Fuel  Limit 

,Lbs.) 

5,000 

5,000 

TABLE  A2.  INSTRUCTIONS  FOR  MANEUVER  I 


1.  Initial  Conditions.  The  threat  aircraft  is  portioned  4,000 
ft  ahead  of  you  at  400  Knots  and  10,000  ft  altitude.  You  are  co-speed 
co-altitude  and  on  the  same  heading.  You  have  a  'full  load  (640  rounds) 
of  20mm,  are  clean,  have  no  missiles,  anu  have  8,000  pounds  of  fuel. 

2.  Target  Maneuvers.  Upon  commencement  of  the  engagement,  the 
threat  aircraft  will  maintain  straight  and  level  at  400  Knots  while  you 
close  to  gun  range.  Three  seconds  after  you  pass  2,500  ft  range  the 

et  will  begin  a  4  g,  full  AB  turn  in  either  direction,  maintaining 
'■•is  for  90  degrees  of  turn.  After  90  degrees,  4  g's  will  be  main¬ 
tained  hut  the  bank  angle  will  be  reduced  to  30  degrees.  The  target 
will  then  be  in  a  climb  maintaining  4  g's  until  300  Knots.  At  300 
Knots,  the  target  will  roll  inverted  and  pull  down  to  the  horizon. 

When  his  caged  sight  line  touches  the  horizon,  he  will  roll  wings  love! 
and  call  cease  maneuver. 

3.  Object  of  Mission.  Attempt  to  hit  the  target  with  20mm  as 
many  times  as  possible.  Are  there  questions? 


INSTRUCTIONS  TO  TARGET  PILOT  | 

(Not  read  to  attacking  pilot)  ! 

Maneuver.  Perform  the  exercise  on  instruments.  The  console  opera¬ 
tor  will  inform  you  when  the  attacker  is  at  2,500  ft.  Wait  3  seconds 
and  start  the  maneuver.  Call  cease  maneuver  when  you  have  rolled 
wings  level  after  your  pull  down  to  the  horizon. 


TABLE  A3.  INITIAL  CONDITIONS  FOR  MANEUVER  II 


Variables 

Units 

Cockpit  1 

Indicated  Airspeed 

(Knots) 

400 

AT  ti tude 

(feet) 

15,000 

Heading 

(degrees) 

0 

Line  of  Sight  Angle 

(degrees) 

30 

Slant  Range 

(feet) 

9,000 

Relative  Horizontal 
Offset 

(feet) 

4,500 

Cannon  Rounds 

(#) 

640 

AIM-9  (Heat)  Missiles 

(#) 

4 

AIM-7  (Radar)  Missiles 

(#) 

0 

ruel  Quantity 

(Lbs.) 

8,000 

Bingo  Fuel  Limit 

(Lbs.) 

5,000 

TABLE  A4.  INSTRUCTIONS  FOR  MANEUVER  II 


1.  Initial  conditions.  The  threat  aircraft  is  positioned  in 
your  forward  quadrant  at  15,000  ft  MSL  and  400  Knots.  You  are  at  30 
degrees  aspect  to  the  target  at  approximately  9,000  ft  slant  rt-ge. 

You  have  a  full  load  of  20mm,  no  tanks,  4  AIM-9Js,  and  8,000  pounds 
of  fuel. 

2.  Target  Maneuvers.  Upon  commencement  of  the  engagement,  the 
threat  aircraft  will  maintain  400  Knots,  straight  and  level  until  you 
are  at  6,000  ft  slant  range.  At  6,000  ft  range,  the  target  will  break 
into  you  using  5-6  G's,  idle  power,  and  speed  brakes.  As  you  over¬ 
shoot  and  begin  a  separation  maneuver,  the  threat  will  reverse  and 
attempt  to  fire  an  AIM-9. 

3.  Object  of  Mission.  Accelerate  and  close  on  the  target  air¬ 
craft.  As  you  approach  6,000  ft  range,  fire  an  AIM-9  at  the  target. 

As  the  target  breaks  into  your  attack,  select  guns  and  set  up  for  a 
high-angle  raking  gun  pass.  Attempt  to  hit  the  target  with  as  many 
20mm  rounds  as  possible  prior  to  initiating  a  separation  maneuver. 
Maneuver  will  cease  when  attacker  has  opened  to  6,000  ft  slant  range. 
Are  there  any  questions? 


INSTRUCTIONS  TO  TARGET  PILOT 
(Not  read  to  attacking  pilot) 


Maneuver.  Perform  the  exercise  on  instruments.  The  console 
operator  will  tell  you  when  attacker  is  at  6,000  ft.  Break  into  the 
attacker.  He  will  be  on  your  left  side.  Perform  an  unloaded  reversal 
when  the  attacker  disappears  behind  you.  Launch  an  AIM-9  if  possible. 
Any  questions? 


TABLE  A5.  INITIAL  CONDITIONS  FOR  MANEUVER  III 


Variables  Units  Cockpit  1  Cockpit  2 


Indicated  Airspeed 

(Knots) 

400 

Altitude 

(feet) 

14,000 

Heading 

(degrees) 

0 

Line  of  Sight  Angle 

(degrees) 

0 

Slant  Range 

(feet) 

Relative  Horizontal 
Offset 

(feet) 

Cannon  Rounds 

(#) 

640 

AIM-9  (Heat  Missiles 

(#) 

4 

AIM-7  (Radar)  Missiles 

(#) 

0 

Fuel  Quantity 

(Lbs.) 

8,000 

Bingo  Fuel  Limit 

(Lbs.) 

5,000 
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TABLE  A6.  INSTRUCTIONS  FOR  MANEUVER  III 


There  will  be  three  successive  engagements  during  this  maneuver. 
The  following  information  is  applicable  to  all  three. 

1.  First  Run.  Initial  Conditions.  The  threat  aircraft  is 
positioned  on  your  nose  at  3  NM,  opposite  heading.  He  is  at  12,000  ft 
MSL  and  450  Knots.  You  should  be  at  14,000  ft  and  400  Knots.  You 
have  4  AIM-9s,  a  full  load  of  20mm,  no  tanks,  and  8, COO  pounds  of 
fuel.  This  will  be  your  starting  condition  for  the  r.ext  3 engagements . 

2.  Second  Run .  Target  Maneuvers.  The  target  will  maneuver 
counter-offensive ly  in  an  unpredictable  manner  on  all  three  runs. 

Take  no  action  until  you  pass  the  target  aircraft.  After  the  initial 
pass  the  target  aircraft  will  fire  on  you  with  AIM-9  missiles  or  guns 
if  you  give  him  the  opportunity  to  do  so. 

3.  Third  Run.  Object  of  the  Mission.  Kill  override  will  be 
off.  The  object  is  to  expeditiously  achieve  a  kill  on  the  threat  with 
the  ordnance  available.  Bingo  is  5,000  pounds.  Call  Bingo.  Are 
there  any  questions? 


INSTRUCTIONS  TO  TARGET  PILOT 
(Not  read  to  attacking  Dilot) 


1.  First  Run.  As  you  pass  the  attacker,  enter  a  4-5g  AB  half 
Cuban  eight.  Accelerate  in  the  45  degree  dive  to  400  Knots,  then  pull 
up  to  enter  a  3  g/500  Knots  level  turn  right.  Do  this  on  instruments. 
Any  questions? 

2.  Second  Run.  Delay  for  5  seconds  after  you  pass  the  attacker 
then  perform  a  4-5g  AB  loop  followed  by  a  3g,  500  Knots  level  turn 
right.  Do  this  on  instruments.  Any  questions? 

3.  Third  Run.  As  you  pass  the  attacker,  initiate  a  3g/450  Knots 
level  turn  left.  Maintain  this  turn  for  20  seconds,  then  reverse  the 
turn  and  continue  at  3g,  450  Knots.  Do  this  on  instruments.  Any 
questions? 

(For  all  three  runs,  the  target  pilot  would  continue  to 
make  alternate  direction  left  and  right  level  turns  un¬ 
til  the  maneuver  ended.) 
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TABLE  A7.  INITIAL  CONDITIONS  FOR  THE  FIRST  OF 
THREE  lvl  FREE  ENGAGEMENT 


Variables 

Units 

Cockpit  1 

Cockpit  2 

Indicated  Airspeed 

(Knots) 

400 

400 

Altitude 

(feet) 

15,000 

15,000 

Heading 

(degrees) 

0 

0 

Line  of  Sight  Angle 

(degrees) 

90  Left 

90  Right 

Slant  Range 

(feet) 

6,000 

Relative  Horizontal 

Offset 

(feet) 

6,000 

Cannon  Rounds 

(#) 

640 

640 

AIM-9  (Heat)  Missiles 

(#) 

2 

2  (4)* 

AIM-7  (Radar)  Missiles 

(#) 

0  (2)* 

0 

Fuel  Quantity 

(Lbs.) 

7,000 

7,000 

Bingo  Fuel  Limit 

(Lbs.) 

4,000 

4,000  (2500)* 

*  For  F-4  vs  MIG  (cockpit  2)  differences  in  initial  conditions  are 
shown  in  parenthesis. 
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TABLE  A8.  INITIAL  CONDITIONS  FOR  SECOND  FREE  ENGAGEMENT 


Variables 

Units 

Cockpit  1 

Cockpit  2 

Indicated  Airspeed 

(Knots) 

450 

350 

Altitude 

(feet) 

15,000 

17,000 

Heading 

(degrees) 

0 

0 

Line  of  Sight  Angle 

(degrees) 

90  Left 

90 

Right 

Slant  Range 

(feet) 

6, 

300 

Relative  Horizontal 

Offset 

(feet) 

6,000 

Cannon  Rounds 

(#) 

640 

640 

(400)* 

AIM-9  (Heat)  Missiles 

(#) 

2 

2 

(4) 

AIM-7  (Radar)  Missiles 

(#) 

0  (2)* 

0 

Fuel  Quantity 

(Lbs.) 

7,000 

7,000 

(4000) 

Bingo  Fuel  Limit 

(Lbs.) 

4,000 

(2500) 

*For  F-4  vs  MIG  (Cockpit  2)  differences  in  initial  conditions  are  shown 
in  parenthesis. 


TABLE  A9.  INITIAL  CONDITIONS  FOR  THIRD  FREE  ENGAGEMENT 


Variables 

Uni  ts 

Cockpit  1 

Cockpit  2 

Indicated  Airspeed 

(Knots) 

400 

400 

A1  ti tude 

(feet) 

15,000 

15,000 

Heading 

(degrees) 

0 

180 

Line  of  Sight  Angle 

(degrees) 

90  Left 

90  Left 

Slant  Range 

(feet) 

2,000 

Relative  Horizontal 

Offset 

(feet) 

2,000 

Cannon  Rounds 

(#) 

640 

640  (400)* 

AIM-9  (Heat)  Missiles 

(#) 

2 

2  (4)* 

AIM-7  (Radar)  Missiles  (#) 

0  (2)* 

0 

Fuel  Quantity 

(Lbs.) 

7,000 

7,000  (4000)* 

Bingo  Fuel  Limit 

(Lbs.) 

4,000 

(2500)* 

For  F-4  vs  MIG  (Cockpit  2)  differences  in  initial  conditions  are  shown 
in  parenthesis. 
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RATING  CRITERIA 

U-  Performance  not  observed  or  insufficent  information  to  make  judgment. 
0-  INDICATES  LACK  OF  ABILITY  OR  KNOWLEDGE. 

1-  INDICATES  LIMITED  PROFICIENCY.  MAKES  ERRORS  OF  OMISSION  OR  COMMISSION 

2-  PERFORMANCE  IS  ESSENTIALLY  CORRECT.  RECOGNI2ES  AND  CORRECTS  ERRORS. 

3-  PERFORMANCE  IS  CORRECT,  EFFICIENT,  SKILLFUL  AND  WITHOUT  HESITATION 
i-  PERFORMANCE  REFLECTS  AN  UNUSUALLY  HIGH  OEGREE  OF  PROFICIENCY. 


Figure  A2.  Pilot  Rating  Form 


APPENDIX  B: 
PRETEST  DATA 


TABLE  Bl.  MANEUVER  III,  TIME  TO  KILL  (SEC)  &  NUMBER  OF  KILLS 


Week  Pilot 


1 

5 

1 

6 

2 

1 

2 

2 

2 

3 

2 

4 

2 

5 

2 

6 

3 

1 

3 

2 

3 

3 

3 

4 

3 

5 

3 

6 

Engagement 

Total 

mm 

1 

2 

3 

Kills 

0 

- 

- 

115 

1 

115 

232 

97 

114 

3 

148 

134 

109 

66 

3 

103 

116 

75 

70 

3 

87 

135 

113 

126 

3 

125 

180 

189 

-» 

2 

185 

- 

- 

- 

C 

- 

- 

191 

153 

2 

172 

- 

- 

- 

0 

- 

35 

74 

46 

3 

52 

187 

45 

132 

3 

121 

152 

— 

1 

152 

- 

- 

56 

1 

56 

126 

- 

46 

2 

86 

120 

149 

- 

2 

135 

85 

64 

120 

3 

90 

38 

40 

30 

3 

36 

120 

79 

97 

112 

2 

2 

116 

88 

- 

- 

• 

0 

- 

- 

189 

118 

2 

154 

198 

125 

95 

3 

139 

43 

167 

105 

3 

105 

73 

122 

122 

95 

2 

2 

108 

98 

110 

- 

117 

2 

114 

166 

174 

111 

3 

150 

41 

• 

90 

2 

66 

43 

95 

2 
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APPENDIX  C: 


DATA  TAPE  CONTENTS,  FORMAT  AND  MEASUREMENT  CALCULATIONS 


Air  combat  aircraft  and  aircrew  performance  data  were  collected 
using  the  SAAC  simulator  at  Luke  Air  Force  Base  for  numerous  one  versus 
one  ACM  engagements  three  days  a  week  for  five  weeks.  This  collection 
effort  resulted  in  15  tapes  containing  the  results  of  over  640  simulated 
engagements.  The  contents  and  the  format  of  the  data  tapes  produced 
during  this  study  by  the  SIGMA  5  computers  (which  control  the  SAAC 
simulator)  are  documented  in  this  section.  Also  contained  in  this 
section  are  details  on  the  further  manipulation  of  the  data  required  to 
perform  timely  analyses  with  the  SIGMA  7  computer  at  the  NAVAL  TRAINING 
EQUIPMENT  CENTER  in  Orlando. 

Organization  of  This  Appendix.  To  aid  information  retrieval,  all  of 
the  figures  and  tables  in  this  appendix  are  at  the  end  of  the  text.  Figures 
Cl  through  C8  appear  first  and  are  followed  by  Tables  Cl  through  C4. 

SAAC  Data  Tapes.  Each  ACM  engagement,  as  it  was  recorded  on  tape, 
consisted  of  one  header  record  and  a  variable  number  of  data  blocks 
followed  by  a  trailer  record  (Figure  Cl).  These  time  histories  con¬ 
tained  a  variable  number  of  data  blocks,  as  the  time  required  to 
complete  an  ACM  engagement  could  not  be  predetermined.  All  of  the 
buffers  or  data  blocks  were  400  words  in  length. 

The  header  record  and  trailer  record  contain  information  specific 
to  the  operation  of  the  SAAC  data  collection  system  and  are  not  directly 
pertinent  to  decoding  the  rest  of  the  data  base.  These  records  were  not 
retained  in  any  further  copy  of  the  data  and  need  not  be  documented  in 
this  report. 

Each  data  record  was  written  at  a  two  hertz  rate  and  contained  both 
two  hertz  and  ten  hertz  parametr 'S .  The  general  structure  of  each  data 
record  is  shown  in  Figure  C2.  ach  record  consists  of  a  record  infor¬ 
mation  section,  the  two  hertz  parameters  and  finally  five  sets  of  ten 
hertz  parameters.  The  data  record  information  section  is  represented  in 
Figure  C3.  Of  primary  importance  are  the  mission  numbers  $md  pilot 
numbers  contained  in  this  data  section.  Following  the  information 
section  (as  can  be  seen  in  Figure  C2)  are  the  actual  values  of  the  two 
hertz  flight  parameters.  These  values  come  in  pairs;  one' for  each 
cockpit.  Each  set  of  ten  hertz  parameters  are  preceded  by  two  informa¬ 
tion  words  (Hertz  count  and  simulator  mode).  There  are  62  pairs  of  two 
hertz  parameters  and  ten  pairs  of  ten  hertz  parameters  contained  in 
their  respective  sections.  More  about  the  specific  parameters  is 
contained  in  a  later  section  of  this  appendix. 


The  data  records  are  written  continuously  across  the  data  blocks 
with  the  first  word  in  each  data  block  being  the  number  of  good  words 
contained  in  that  respective  block.  Since  the  data  record  size  is  not 
related  to  the  block  size,  the  location  of  any  particular  parameter  is 
not  fixed  in  every  data  block.  The  data  must,  therefore,  be  uncoded/ 
unpacked  relative  to  the  start  of  the  first  data  block  in  the  engage¬ 
ment.  The  data  block  structure  imposed  on  the  records  is  illustrated 
in  Figure  C4. 

New  Header  Blocks.  Any  further  analysis  of  the  ACM  data  required 
the  inclusion  of  engagement  outcome  (kill  data)  or  pilot  questionnaire 
data.  To  this  end  a  new  header  block  structure  was  developed  (Table 
Cl).  Kill  data  and  questionnaire  results  were  punched  on  cards  and  then 
merged  with  other  miscellaneous  information  on  tape  to  produce  a  new 
header  block  for  each  ACM  engagement. 

Analysis  Generation.  In  one  pass  through  the  15  SAAC  data  tapes 
one  summary  data  tape  and  seven  compressed  data  tapes  were  produced. 

The  summary  data  tape  is  a  merger  of  the  new  header  blocks  and  summary 
statistics  computed  on  each  engagement.  This  tape  is  useful  for  rapid 
glances  at  selected  mission  success  factors.  The  compressed  data  tapes 
are  a  merger  of  the  header  blocks  and  reblocked  two  hertz  data  records. 
The  original  header  and  trailer  records  were  dropped  and  the  two  hertz 
data  recordi  reorganized  into  blocks  containing  30  second  multiples. 

This  reblocking  simplifies  the  extraction  of  selected  parameters  for 
tactical  analysis  and  reduces  the  number  of  tapes  to  be  handled  in  any 
analysis.  The  generation  of  these  new  tapes  is  illustrated  in  Figure 
C5. 

Before  describing  the  final  tape  structures,  it  is  important  to 
mention  that  some  data  editing  was  performed  in  the  process  of  generating 
these  last  tapes.  Data  recording,  being  a  manual  function,  was  not 
always  synchronized  with  the  release  of  the  aircraft  at  the  start  of  an 
engagement  and  with  the  termination  of  the  engagements.  Therefore, 
useless  data  appears  at  the  beginning  and  end  of  most  of  the  engagements 
on  tape.  There  were  also  false  starts  where  recording  began  but  the 
engagement  had  to  be  reinitialized  and  then  restarted.  Compression  of 
the  data  and  computation  of  summary  statistics  could  not  begin  until  the 
detection  and  removal  of  these  problematic  data.  Logic  was  implemented 
to  detect  the  release  of  the  aircraft  at  the  beginning  of  an  engagement 
by  checking  for  consistent  changes  in  the  ground  track  of  both  aircraft. 
If  either  aircraft  struck  the  ground,  ran  out  of  fuel,  stopped  moving, 
or  was  reset,  the  engagement  was  considered  terminated.  With  these 
boundaries  defined,  the  engagement  also  had  to  last  over  10  seconds 
before  it  was  considered  good  data  and  included  in  the  generation 
process . 
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Data  Summary  Tape  Structure.  Each  of  the  172  parameters  (86  para- 
meters  for  2  pilots)  are  represented  in  the  DATA  SUMMARY  BLOCK.  To 
access  the  summary  data  an  integer  block  2500  words  long  must  be  read 
and  equivalenced  to  a  real  array  250  x  10.  The  first  dimension  of  the 
array  corresponds  to  the  variable  number  or  location  found  in  the 
compressed  data  block.  The  second  dimension  corresponds  to  each  of  the 
measure  transforms.  In  other  words,  there  are  10  transforms  for  each 
of  172  parameters  on  store  for  each  mission.  Also,  on  the  tape  can  be 
found  corresponding  HEADER  BLOCKS  with  other  related  information  about 
each  run.  The  HEADER  BLOCK  is  represented  on  the  tape  as  entirely 
integer  and  160  words  long.  It  also  must  be  equivalenced  to  a  real 
array  to  access  the  floating  point  portions.  The  summary  data  tape 
structure  is  illustrated  in  Figure  C6.  The  statistical  transforms  are 
contained  in  Table  C5. 


Raw  Data  Tape  (Compressed)  Structure.  Each  mission  record  is  pre- 
ceded  by  the  same  HEADER  BLOCK  previously  described.  These  data  are 
then  stored  in  integer  form  in  30  second  multiples  of  214  in  blocks,  a 
maximum  of  7000  words  long,  for  each  complete  mission.  The  last  block 
may  be  less  than  30  seconds  (i.e.,  6420  words)  long  depending  on  mission 
duration.  Each  entire  mission  is  followed  by  one  EOF  mark.  To  access 
data  in  the  DATA  BLOCKS,  like  all  the  others,  it  must  be  equivalenced  to 
a  real  array.  The  compressed  data  structure  is  illustrated  in  Figure 
C7  and  the  contents  are  described  in  Tables  C2  and  C3. 
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HEADER  RECORD 


DATA  BLOCK  1 


DATA  BLOCK  2 


TRAILER  RECORD 


Figure  Cl.  SAAC  Data  Tape  S 
for  Entire  Engagemen 


Header 

Block 


Run  1 


TABLE  Cl.  HEADER  BLOCK  CONTENTS 


Origin^ 

Word 

Contents 

Char.  Type 

1 

1 

RUN  NUMBER 

I 

1 

2 

PROBLEM  SET-UP  NUMBER 

I 

1 

3-4 

PILOT  NAME  COCKPIT  1  (AT  RUNTIME) 

A 

1 

5-6 

PILOT  NAME  COCKPIT  2  (AT  RUNTIME) 

A 

2 

7 

LENGTH  OF  RUN  (NO.  OF  1/2  SEC  FRAMES) 

I 

- 

8-10 

UNUSED 

- 

3 

11 

PILOT  KILLED  1,2,3  (BOTH) 

I 

3 

12 

AIM  9  (0-1 ) 

I 

3 

13 

AIM  7  (0-1) 

I 

3 

14 

GUN  (0-1) 

I 

3 

15 

GROUND  (0-1) 

I 

3 

16 

•G's  (0-1) 

I 

3 

17 

BINGO  (0-1) 

I 

3 

18 

COCKPIT  1  PILOT  NO. 

I 

3 

19 

COCKPIT  1  WK  NO. 

I 

3 

20 

COCKPIT  2  PILOT  NO. 

I 

3 

21 

COCKPIT  2  WK  NO. 

T 

* 

3 

22 

ERROR  CODE  (O-PRF.,  1-NORM,  2-DSM, 

9-ERR,  7-DROP) 

I 

4 

23-24 

COCKPIT  1  PILOTS  NAME 

A 

4 

25-26 

COCKPIT  1  PILOTS  SQUADRON 

A 

4 

27-28 

COCKPIT  1  PILOTS  WING 

A 

4 

29 

AGE  PILOT  1 

I 

4 

30 

TOT.  HR 

I 

4 

31 

TOT.  FIGHTER  HR. 

I 

4 

32 

F4  FRONT  HR. 

I 

4 

33 

F4  BACK  HR. 

I 
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TABLE  Cl.  HEADER  BLOCK  COPTENTS  (Cont.) 


Origin^ 

Word 

Contents 

Char.  Type 

4 

34 

TOT.  HR.  LAST  6  MO. 

PILOT  1 

I 

4 

35 

TOT.  F4  HR.  LAST  6  MO. 

11 

I 

4 

36 

TOT.  SIM.  HRS.  LAST  6  MO. 

11 

'  I 

4 

37 

ACM  ENGAGEMENTS 

i: 

I 

4 

38 

ACME  LAST  6  MG. 

it 

I 

4 

39 

A/A  SIM  (0-1) 

ii 

I 

4 

40 

A/A  SIM  LAST  6  MO. 

u 

I 

4 

41 

DACT  (0-1) 

ii 

I 

4 

42 

DACT  LAST  6  MO. 

it 

I 

4 

43 

SAAC  TIME 

ii 

I 

4 

4 

COMBAT  HRS. 

ti 

I 

4 

45 

COMBAT  ENGAGEMENTS 

ii 

I 

4 

46 

FWIC  GRADUATE  (0-1) 

ii 

I 

4 

47 

FWIC  RECENCY 

it 

I 

4 

48 

TAC  ACES  (0-1) 

n 

I 

4 

49 

DACT  RECENCY 

ii 

I 

4 

50 

TAC  ACES  RECENCY 

ii 

I 

4 

51 

FWS  AGRESSOR  (0-1) 

ii 

I 

4 

52 

ACM  INSTRUCTOR  (0-1) 

ii 

I 

4 

53 

AIM-7  FIRED 

ii 

I 

4 

54 

AIM-9  FIRED 

ii 

I 

4 

55 

TCM  3-1  RECENCY 

ii 

I 

4 

56 

HEIGHT 

ii 

I 

4 

57 

WEIGHT 

ii 

I 

4 

58 

SPORTS  (0-1) 

it 

I 

4 

59 

PHYS  COND.  (0-1) 

it 

I 

L*3!d  ’rfi-,  x<u„ 


s’i. 


7*7?. #T^v^" 


w^-x  v-ytw 


TABLE  Cl.  HEADER  BLOCK  CONTENTS  (Cont.) 


Origin1 

Word 

Contents 

Char.  Type 

4 

60 

OTHER  PILOTS  (0-1) 

PILOT  1 

I 

4 

61 

PILOTS  NO. 

II 

I 

4 

62 

WEEK  NO. 

II 

I 

- 

63 

UNUSED 

II 

- 

5 

64 

PEER  RANK 

II 

F 

5 

65 

PROJ.  PILOT  RANK 

II 

F 

5 

66 

AVG.  DECISIVENESS 

II 

F 

5 

67 

AVG.  SIT.  AWARENESS 

II 

F 

5 

68 

AVG.  MANEUVERING 

II 

F 

5 

69 

AVG.  AGGRESSIVENESS 

II 

F 

5 

70 

AVG.  PERFORMANCE 

II 

F 

4 

71-72 

COCKPIT  2  PILOTS  NAME 

A 

4 

73-74 

COCKPIT  2  PILOTS  SQUADRON 

A 

4 

75-76 

COCKPIT  2  PILOTS  WING 

A 

4 

77 

AGE 

PILOT  2 

V 

l 

4 

78 

TOT.  HR. 

II 

I 

4 

79 

TOT.  FIGHTER  HR. 

II 

I 

4 

80 

F4  FRONT  HR. 

II 

I 

4 

81 

F4  BACK  HR. 

II 

I 

4 

82 

TOT.  HR.  LAST  6  MO. 

It 

I 

4 

83 

TOT.  F4  HR.  LAST  6  MO. 

II 

I 

4 

84 

TOT.  SIM.  HR.  LAST  6  MC. 

II 

I 

4 

85 

ACM  ENGAGEMENTS 

II 

I 

4 

86 

ACME  LAST  6  MO. 

II 

I 

4 

87 

A/A  SIM  (0-1) 

II 

I 

4 

88 

A/ A  SIM  LAST  6  MO. 

II 

I 

4 

89 

DACT  (0-1) 

II 

I 

TABLE  Cl  HEADER  BLOCK  CONTENTS 

(Cont.) 

Word 

Content.r 

Char.  Type 

90 

DACT  LAST  *5  MO. 

PILOT  2 

I 

91 

SAAC  TiKS 

It 

I 

92 

COMBAT  HR:,. 

II 

I 

93 

COMBAT  ENGAGEMEU,  * 

II 

I 

94 

FWIC  GRADLViC  (0*1} 

II 

I 

96 

FWIC  RECENCY 

It 

I 

96 

TAC  AC-.  (0-1) 

II 

I 

97 

DAC'i  KllZNCY 

II 

I 

V3 

TAC  ACt\  RECENCY 

II 

I 

99 

FWS  AGGRESSOR  (0-1) 

II 

I 

*  -N  r 

lu,l 

ACM  iNSTPUC  \  .  (0-1) 

II 

I 

101 

AIM-7  FIREC 

II 

y 

102 

AiM-9  FIRED 

II 

I 

103 

TCM  3-1  RECENCY 

II 

I 

104 

HEIGHT 

II 

I 

105 

WEIGHT 

II 

I 

106 

SPORTS  (0-1) 

II 

I 

107 

PHYS  COND.  (0-1) 

II 

I 

108 

OTHER  PILOTS  (0-1) 

It 

I 

109 

PILOTS  NO. 

II 

I 

110 

WEEK  NO. 

l( 

I 

111 

UNUSED 

II 

I 

112 

PEER  RANK 

II 

F 

113 

PROJ.  PILOT  RANK 

11 

F 

111 

AVG.  DECISIVENESS 

II 

F 

115 

AVG.  SIT.  AWARENESS 

II 

F 

116 

AVG.  MANEUVERING 

II 

F 

TABLE  Cl.  HEADER  BLOCK  CONTENTS  (Cont.) 


Word  Contents 


117  AVG.  AGGRESSIVENESS  PILOT  2 

118  ABG.  PERFORMANCE 


Key  to  Origin 

1.  Original  SAAC  Data 

2.  Calculated  during  data  editing  phase 

3.  Kill  data  cards 

4.  Pilot  questionnaires 

5.  Peer  ranking  questionnaires 


TABLE  C2.  COMPRESSED  DATA  BLOCK  STRUCTURE 


Words 

Parameters 

Sampling  Rate^ 

Unit 

1-2 

NORTH 

FT. 

3-4 

EAST 

FT. 

5-6 

ALTITUDE 

FT. 

7-8 

ALT  RATE 

FT/SEC 

9-10 

NAV  PITCH 

+  180° 

11-12 

NAV  ROLL 

+  180° 

13-14 

NAV  HDG 

+  180° 

15-16 

SLANT  RANGE 

FT. 

17-18 

19-20 

qq£E  j  LOS  AZIMUTH 

21-22 

23-24 

j  LOS  ELEVATION 

: 

25-26 

27-28 

cosE  )  L0S  R0LL 

- 

29-30 

OUT  OF  VIEW 

LOGICAL 

31-32  ' 

A/C  HEG 

DEG. 

33-34 

A/C  PITCH 

DEG. 

35-36 

A/C  ROLL 

DEG. 

37-38 

SIDE  SLIP 

DEG. 

39-40 

AOA 

UNITS 

41-42 

AIRSPEED 

KNTS 

43-44 

COEF  LIFT 

- 

45-46 

TURN  RATE 

RAD/ SEC 

47-48 

•G' 

- 

49-50 

LONG  ) 

FT. /SEC 

51-52 

LAT  )  A/C  AXIS  VEL 

FT. /SEC 

53-54 

VERT  ) 

FT. /SEC 
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TABLE  C2.  COMPRESSED  DATA  BLOCK  STRUCTURE  (Cont.) 


•a m 

‘  .3 


Words 

Parameters 

Sampling  Rate1  Units 

55-56 

SPEED  BRAKES 

POSITION 

57-58 

THROTTLE  LEFT 

POSITION 

59-60 

THROTTLE  RIGHT 

POSITION 

61-62 

INTERNAL  FUEL 

LBS. 

63-64 

FUEL  FL0W-L 

LBS. /HR. 

65-66 

FUEL  FL0W-R 

LBS. /HR. 

67-68 

PITCH  TRIM 

LOGICAL 

69-70 

H0RIZ  TRIM 

LOGICAL 

71-72 

OFF  ) 

LOGICAL 

73-74 

BST  ) 

SELECT  (INST.  CONSOLE)  LOGICAL 

75-76 

RDR  ) 

LOGICAL 

77-78 

AUTO  RNG  ] 

i 

LOGICAL 

79-80 

rng  i  ; 

t 

LOGICAL 

81-82 

RNG  2  : 

\ 

LOGICAL 

83-84 

RNG  3  j 

RDR  RNG  ] 

LOGICAL 

85-86 

RNG  4  ] 

\ 

i 

LOGICAL 

\ 

i 

INST.  CONS 

87-88 

TGT  ON 

\ 

l 

LOGICAL 

89-90 

AUTO  LOCK 

LOGICAL 

91-92 

AUTO  ACQUISITION  ] 

LOGICAL 

93-94 

MASTER  ARM  ) 

LOGICAL 

95-96 

GUN  SELECT  ) 

LOGICAL 

97-98 

GUN/STORE  SELECT)  TOGGLES  LOGICAL 

99-100 

NOSE  SELECT 

LOGICAL 

101-102 

GUN  HIGH  RATE  ) 

LOGICAL 

103-104 

PWR  H  ) 

RADAR  PWR 

LOGICAL 

105-106 

PWR  E  ) 

LOGICAL 

o  o 

VO  "--4 

1  1 
■  i  >  1 1 

— *  o 

O  00 

MMRJ  J  RADAR/HEAT/REJECT  jS 

111-112 

INTERLOCK  OUT  ) 

LOGICAL 

113-114 

MASTER  ARM  )  TOGGLES  LOGICAL 

115-116 

TRIGGER 

) 

LOGICAL 

1 


TABLE  C2.  COMPRESSED  DATA  BLOCK  STRUCTURE  (Cont.) 


Words 

Parameters 

Sampling  Rate' 

Uni  ts 

117-118 

AIM-7  FIRED 

119-120 

AIM-9  FIRED 

- 

121-122 

GUNS 

- 

123-124 

GUN  HITS 

- 

125-1262 

LOo  AZIMUTH  ANCLE 

RAD. 

127-1282 

L :  -.lEVATION  ANGLE 

RAD. 

129-1302 

LOS  GRAND  ANGLE 

RAD. 

131-1322 

ASPECT  GRAND  ANGLE 

RAD. 

133-1342 

ENERGY  MANAGEMENT  INDEX 

- 

135-1362 

CLOSING  RATE 

FT. /SEC 

137-1382 

OFFENSE 

LOGICAL 

139-1402 

DEFENSE 

LOGICAL 

141-1422 

SADDLE 

LOGICAL 

143-1442 

HEAD  ON 

LOGICAL 

145-1462 

TAC  ACES 

- 

147-1482 

THROTTLE  POS 

UNITS 

149-1502 

LEADING 

LOGICAL 

151-1522 

IN  RANGE 

LOGICAL 

153-1542 

ROLL  RATE 

DEG/SEC 

155-1562 

(ROll  RATE  X  ALT  RATE) 

- 

157-1582 

PLANE  OF  ACTION 

DEGREES 

159-1602 

DEFENSIVE  W/ DISADVANTAGE 

LOGICAL 

161-1622 

A0A>28° 

LOGICAL 

163-1642 

HEADING  RATE 

DEG/SEC 

165-166 

FORCE  )  PITCH 

These  measures 

LBS. 

167-168 

POSITION  ) 

were  sampled 

RAD. 

169-170 

FORCE  )  AILERON 

at  10  Hz. 

LBS. 

171-172 

POSITION  ) 

RAD. 

,? 

•J 

1 

•| 


TABLE  C 2.  COMPRESSED  DATA  BLOCK  STRUCTURE  (Cont.) 


Words 

Parameters 

Sampling  Rate^ 

Units 

173-174 

POSITION 

RUDDER 

RAD. 

175-176 

FORCE  ) 

PITCH 

These  measures  were 

LBS. 

177-178 

POSITION  ) 

sampled  at  10  Hz. 

RAD. 

179-180 

181-182 

FORCE  ) 

POSITION  ) 

AILERON 

LBS. 

RAD. 

183-184 

POSITION 

RUDDER 

RAD. 

185-186 

FORCE  ) 

PITCH 

II  II  II  II 

LBS. 

187-188 

POSITION  ) 

RAD. 

189-190 

FORCE  ) 

AILERON 

LBS. 

191-192 

POSITION  ) 

RAD. 

193-194 

POSITION 

RUDDER 

RAD. 

195-196 

FORCE  ) 

PITCH 

It  II  II  II 

LBS. 

197-198 

POSITION  ) 

RAD. 

199-200 

FORCE  ) 

AILERON 

LBS. 

201-202 

POSITION  ) 

RAD. 

203-204 

POSITION 

RUDDER 

RAD. 

205-206 

FORCE  ) 

PITCH 

11  II  II  II 

LBS. 

207-208 

POSITION  ) 

RAD. 

209-210 

FORCE  ) 

AILERON 

LBS. 

211-212 

POSITION  ) 

RAD. 

213-21.4 

POSITION  ) 

RUDDER 

RAD. 

4 

a 


1 


:! 


a 

**3 

i 

l 

■J 


•J 

J.C 

*S 

1 

n: 

>] 

"ja 

4« 

I 

\§ 

4 


*3 

,VS 

JS 

1 

*$ 

J2 

ija 


1  All  of  the  measures  were  sampled  at  2  Hz.,  unless  otherwise  stated. 


2  These  measures  were  calculated  from  other  available  measures  or 
parameters  during  the  data  verification  phase  of  the  program. 


im 


3 

- 

'J 

vl 


Wfe~ 


TABLE  C3.  CALCULATED  VARIABLES  ADDED  TO  COMPRESSED  OATA  BLOCK 


Variable  Lo.  Name  Calculation 


125-126 

LOS  AZIMUTH  ANGLE 

ATAN2  (SINE  LOS  AZ,  COS  LOS  AZ) 

127-128 

LOS  ELEVATION  ANGLE 

ATAN2  (SINE  LOS  EL,  COS  LOS  EL) 

129-130 

LOS  GRAND  ANGLE 

ACOS  (COS  LOS  AZ)  (COS  LOS  EL) 

131-132 

ASPECT  GRAND  ANGLE 

180O-0PP0NENT'S  LOS  GRAND  ANGLE 

133-134 

ENERGY  MANAGEMENT  INDEX 

(Fuel 

Remaining-Bingo  Fuel)  ( 

2G  Altitude  +  V^  1 

(Fuel 

Flow  (Left)  +  Fuel  Flow  (Right))At 

Where : 

At  is  015  seconds, 

V  is  velocity  in  ft. /sec. 

135-136 

CLOSING  RAGE 

ASlant  Range/Sec. 

137-138 

' OFFENSE 

LOS  CONE  <  60° 

139-140 

DEFENSE 

LOS  CONE  >  120° 

141-142 

SAOOLE 

OFFESNIVE,  ASPECT  <  90° 

143-144 

HEAD  ON 

BOTH  LOS  <30° 

145-146 

TAC  ACES 

N/A 

147-148 

THROTTLE  TRANSITIONS 

INTO: 

Zone  1 

0<  Throttle  Setting  <  35  Units 

Zone  2 

35<  Throttle  Setting  <  80  Units 

Zone  3 

8Q<  Throttle  Setting  <105  Units 

Zone  4 

105<_  Throttle  Setting 

149-150 

LEADING 

LOS  <  60°,  ASPECT  <  90°, 

LOS  <  ASPECT 

151-152 

IN  RANGE 

LOS  <  60°,  ASPECT  <  90°, 

RANGE  <2000  ft. 

153-154 

ABSOLUTE  ROLL  RATE 

A  [ROLL  ANGLE/SEC! 

155-156 

ABSOLUTE  ROLL  RATE 

X  ALT  RATE 

{ROLL  RATE  X  ALT  RATE', 

157-158 

PLANE  OF  ACTION 

(See  TABLE  C4) 

159-160 

DEFENSIVE  w/DISADVANTAGE  L0S>120,  ASPECT>90,RANGE<4000  ft. 

161-162 

A0A>28° 

A0A>28° 

163-164 

HEADING  RATE 

AHEADING/SEC. 

29-30 

OUT  OF  VIEW  (00V) 

(See  Figure  C8). 

TABLE  C4.  PLANE  OF  ACTION  CALCULATION 


To  determine  the  plane  of  action  (PCA)  for  the  a/c  it  was 
necessary  to  convert  the  body  axis  to  the  inertial  axis  using  pitch 
(o)  and  roll  (j>).  Yaw  was  held  zero  for  convenience  since  any 
yawing  action  would  be  entirely  contained  in  the  POA.  The  body  to 
inertial  transformation  matrix,  R,  is: 


cose 

0 

-sin6 


sin6sin$ 

COSy 

cosSsinO 


sinecos$ 

-sir.o 

cos6cos$ 


The  equation  of  the  plane  (obc)  through  (0,0,0),  (b^bg.bj)  and 
(c^Cj.Cj)  is  Ax-8y+Cz=0, 


Where  A  =  b2  b3 
c2  c3 


=  (b2c3.b3c2), 


3  =  bl  b3  =  (b1c3-b3c1 ) , 

C1  C3 

and  C  =  b]  b2  |  .  (b^z-bzc}). 

C1  C2  | 

The  angle  between  obc  and  the  horizontal  plane  Z  =  -1  is 


But  if  C2<'1,  then  the  a/c  is  inverted  and  }  becomes 
The  final  equation  becomes 


tan  i  (sinecoso)^  +  sW 


cosScoso 


J 


TABLE  C5 .  TRANSFORMATIONS  USED  FOR  SUMMARY  DATA  STATISTICS 


Transforms  on  each  parameter  in  Summary  Data  Block 


1. 

MEAN 

LA 

n 

2. 

S.D. 

- 

EX2- 

■(H)2 

n 

H. 

k 

n-1 

3. 

MIN 

- 

Minimum  of  X 

4. 

MAX 

- 

Maximum  of  X 

5. 

RMS 

- 

r,x2l 

n 

k  » 

H. 

6. 

C.P. 

- 

1  -  Lowpass/Highpass 

7. 

INIT 

- 

Initial  Value  of  X 

3. 

FINAL 

- 

Final  Value  of  X 

9 

AAV 

- 

2  |X| 

n 


10.  RANGE 


MAX  -  MIN 


APPENDIX  D: 

RANKING  AND  RATING  DATA  FROM  FREE  ENGAGEMENTS 
TABLE  D1 .  EXPERT  RANKINGS  AND  PEER  RANKINGS 


Pilot 

Meek  I 

r  i  lUu 

1 

2 

3 

4 

5 

1 

6  (5)* 

6  (5.5) 

5  (6) 

4  (4) 

3.5  (6) 

2 

5  (6) 

5  (5.5) 

3  (3.5) 

5  (6) 

3.5  (3) 

3 

3  (3) 

4  (4) 

6  (3.5) 

6  (5) 

5  (4) 

4 

4  (4) 

3  (3) 

4  (5) 

3  (3) 

6  (5) 

5 

1  0) 

1.5  (2) 

1  (1) 

2  (1.5) 

2  (2) 

6 

2  (2) 

1.5  (1) 

2  (2) 

1  (1.5) 

1  (1) 

=  Rank  by  Expert  (Rank  by  Peers). 


TABLE  D2.  EXPERT  RATINGS 


Decis- 
i veness 

Situation 

Awareness 

Maneuvering 
Skill  . 

Aggress¬ 

iveness 

Overall 

Performance 

Group  1 

2.05* 

1.97 

2.05 

2.32 

2.04 

Group  2 

2.13 

2.11 

2.15 

2.30 

2.14 

Group  3 

2.81 

2.85 

2.84 

2.95 

2.84 

*Mean  ratings 
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APPENDIX  E: 

SUPPORTING  DATA  FROM  MULTIVARIATE  ANALYSES 


m 

oo 

r— 

CD 

i-» 

■ — 

ID 

<3* 

M- 

00 

CM 

CM 

cn 

LO 

CM 

U1  N  lO  Ifl 


LO 

o 

00 

CO 

o 

LO 

o\ 

CO 

LO 

CM 

CO 

oo  n 

J _ 

00 

«3* 

00 

CO  r— 

co 

to 

o 

VO 

r—  O 

r—  CO 

o 

CD  <*• 

o 

r—  r— 

LO 

oo 

o 

<*• 

r—  LO 

CM 

o 

o  o 

o  -m- 

CM 

O 

CO 

CL  to 

•  • 

• 

♦ 

ft 

• 

•  • 

• 

• 

•  ft 

•  • 

• 

•  # 

• 

3  C 

O  <B 
s_  <u 

cn  s: 

CM 

CM 

CM 

CM 

CM 
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APPENDIX  F 


TEST  OF  MULTIVARIATE  MODEL  WITH  SIMULATED  DATA 


The  resulting  discriminant  models  were  tested  with  Monte  Carlo  type 
of  data  simulations  to  explore  its  relative  and  potential  reliability. 
Simulations  were  performed  on  the  13  measure  model  for  4  values  of  K  and 
unit  scaling.  The  results  for  the  13  measure  model  are  detailed  in 
Table  FI.  A  simulation  was  performed  by  weighting  the  ACM  data  by  the 
discriminant  coefficients  to  produce  two  distributions.  These  distri¬ 
butions  represented  the  two  groups  in  discriminant  space.  To  test  the 
discriminating  capability  of  the  model  a  break-even  point  was  determined. 
This  break-even  point  is  the  position  on  the  discriminant  axis  (Y)  where 
the  selection  error  was  minimum.  The  average  predicted  selection  error 
can  be  determined  by  dividing  by  two  the  total  observations  that  lie  on 
the  wrong  side  of  the  break-even  point. 

A  test  sample  of  a  thousand  observations  for  each  group  was  gener¬ 
ated.  These  simulated  observations  had  a  random  normal  distribution  with 
the  same  means  and  variance-covariance  matrices  as  the  group  they  were 
intended  to  simulate.  Although  they  were  not  precisely  similar  to  the 
actual  sample  data  or,  probably,  the  real  population,  they  did  provide 
an  independent  measure  of  the  discriminating  ability  of  the  performance 
measurement  models.  This  simulated  data  was  then  weighted  by  the 
discriminant  coefficients  and  the  average  simulated  total  misclassifi- 
cations  were  calculated  for  each  model  using  the  break-even  points. 

The  reduction  in  simulated  misclassifications  by  increasing  K  as 
observed  in  Table  FI  along  with  the  reduction  in  overly  large  coefficients 
(as  observed  in  the  coefficient  trace  in  Figure  5)  is  a  demonstration  of 
the  reduction  of  overfit  with  a  ridge  adjustment.  Further,  it  can  be 
seen  that  the  model  discriminates  better  for  particular  values  of  K.  The 
discriminant  model  outperformed  its  respective  unit  scaling  equation. 

Figures  FI  through  F3  are  computer  frequency  plots  of  the  measure¬ 
ment  models  in  Table  FI.  The  trend  of  improvement  of  the  distributes 
with  the  increase  in  K  was  reaffirmed  when  looking  at  these  plots.  .Note 
the  reduction  in  skewness  and  kurtosis  in  Table  FI  also.)  An  improvement 
in  group  distribution  separation  would  not  be  expected  when  looking  at  a 
ridge  distortion  of  the  original  ACM  data.  Logically,  the  improvement 
in  group  separation  would  be  evident  if  the  series  of  coefficients  could 
be  tried  on  another  sample  from  the  same  population. 


To  look  at  the  discriminant  models  differently,  another  set  of 
simulated  samples  was  generated  as  previously  described.  These  were 
weighted  with  the  coefficients  from  the  13  measure  solutions  to  form 
new  group  distributions.  These  new  distributions  are  described  in  Table 
F2.  As  expected,  the  average  total  percent  misses  predicted  decreased 
as  K  increased.  Quite  visibly,  the  statistical  differences  in  the  two 
distributions  became  larger.  Figure  F4  and  F5  are  frequency  distribution 
plots  of  the  simulated  samples.  An  improvement  in  the  group  separation 
as  K  increases  is  visibly  evident.  Both  the  classification  of  simu¬ 
lated  test  data  and  the  injection  of  simulated  data  into  the  discriminant 
model  to  estimate  predicted  missclassification  errors  are  two  ways  to 
demonstrate  the  reliability  of  various  measurement  models. 
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BLE  F2.  GROUP  DISTRIBUTION  STATISTICS  FOR  SIMULATED  POPULATIONS  IN 
DISCRIMINANT  SPACE  AM  PREDICTED  MISCLASSIFICATIONS. 
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APPENDIX  6 
TACSPACE  DATA 


TABLE  G1 .  80XES  USED  BY  WINNERS  IN  TERMS  OF 
PERCENTAGE  MISSION  TIME 
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28 
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121 

27 
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25 

w 

24 

98.4 

123 

o 

23 

22 

21 

97.6 

132 

222 
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19 
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17 

UJ 

16 

l/) 

UJ 
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O 

ae. 

14 

“  96.0 

in 
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13 

o  94.4 

133 

232 

Note : 

232  of  miss  ion 

time 

12 

Z 
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in  the 
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2  92.8 
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10 

= 
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h- 
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9 

3 
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7 

5  89 
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151 

to 

6 

36 
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231 
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332 

£ 

5 

83 
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142 
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234 

u. 

322 

343 

422 

423 

i 

74 

212 

125 

323 

342 

3 

71 

135 

331 

211 

213 

224 

242 

334 

335 

352 

424 

432 

144 

Ui 

2 

62 

141 

145 

244 

235 

251 

252 

253 

255 

312 

321 

324 

325 

345 

351 

412 

413 

433 
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441 

521 

112 

1 

45 

241 

311 

313 

3U 

341 

344 

353 

354 

411 

421 

431 
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214 
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245 
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414 

0 

28 

315 

355 

415 
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435 

443 

444 

445 

451 

452 

453 

454 
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513 
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515 
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531 
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534 
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541 
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544 
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553 

554 

555 

rwrr-lirrc— i*1 
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33 

32 

31 

30 

29 

23 

39.2 

27 

26 

_  25 
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-  22 

x  21 

97.6 
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UJ 

£  16 

X 
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a  95.2 

ac  ]4 

■— i  1 

U- 

O  „ 

s  13 

z  94.4 

12 

S  92.3 

£  11 

5  92.0 

~  10 

2  91.2 

3 

2  90.4 

H  8 

•-o 

£  1 

°  88 

x  5 

36 

v.  5 

81 

o 
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7? 

8  3 

69 

OS 

£  2 

53 

1 

41 

0 

26 

23%  of  mission  time 
was  spent  in  the 
boxes  above  dotted 


TABLE  G3.  RECIPROCAL  RELATIONSHIPS  OF  TOP  FOURTEEN  BOXES 


RANK 

BOX  CODES 

FOR 

LOSERS 

BOX  CODES 
FOR 

WINNERS 

RANK 

1 

452  ◄ - 

- ►  122 

1 

2 

451  4 - 

- P  121 

2 

3 

453  4 - 

- ►  123 

3 

4 

352  ◄ - 

- *  132 

4 

5 

442  4 - 

- ►  222 

5 

6 

551  4 - 

- ►  111 

6 

7 

342 

7 

8 

353 

133 

8 

9 

552  ^ 

232 

9 

10 

152  4 - 

- ►  152 

10 

11 

343  4 - 

- ►  233 

11 

12 

553  4 - 

- ►  113 

12 

13 

443  ◄ - 

- ►  223 

13 

14 

454  4 - 

- b  124 

14 

Note:  Arrows  show  boxes  which  correspond  by  winner's 
and  loser's  point  of  view,  i.e.,  Box  122  L0S*1 
Aspect*2  Range*2  is  winner's  point  of  view  for- 
same  relationship  between  airplanes  as  Box  452 
L0$=4  Aspects  Range=2  from  loser's  point  of 
view. 
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